A Reddit user on r/MachineLearning has identified a potential issue with the commonly used default learning rate of 2e-4 for QLoRA fine-tuning, particularly when working with smaller datasets (under 10,000 samples). The user found that this default setting can lead to overfitting and poor evaluation results, while a lower learning rate (e.g., 1e-4) with more epochs significantly improved performance on their experiments. They suggest that this default, often cited in tutorials and documentation, may be based on larger datasets and is being blindly copied by users with smaller datasets, leading to wasted time and suboptimal results. AI
IMPACT Highlights a potential pitfall in fine-tuning common models, suggesting a need for more nuanced default parameters for smaller datasets.
RANK_REASON User-generated commentary on a technical detail of a machine learning technique.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
