Hyperfitting

LLM hyperfitting distinct from temperature scaling, study finds

Researchers have identified a phenomenon called hyperfitting in large language models, where fine-tuning on small datasets surprisingly improves generation quality and reduces repetition. This paper demonstrates that hy…