Hyperfitting

New research defines 'Hyperfitting' in LLMs, distinct from temperature scaling

A new research paper introduces the concept of "Hyperfitting," a phenomenon where fine-tuning large language models on small datasets surprisingly improves generation quality and reduces repetition. The study demonstrat…