A new research paper investigates methods to predict when updates to large language models (LLMs) might cause regressions, where a previously correct output becomes incorrect. The study compares various signals, including confidence scores, logit margins, and attention entropy, against cross-version signals like output KL divergence and representation drift. Findings indicate that the effectiveness of these signals is highly dependent on the specific task and model update, with no single signal proving universally superior. However, cross-version signals can remain informative even when confidence metrics fail, suggesting potential for selective fallback mechanisms to older model versions for high-risk samples. AI
IMPACT Provides insights for practitioners to manage LLM updates and mitigate potential regressions by selecting appropriate predictive signals.
RANK_REASON Research paper published on arXiv detailing methods to predict LLM regression.
- alphaXiv
- arXiv
- CatalyzeX
- DagsHub
- Gotit.pub
- Hugging Face
- McQueen
- ScienceCast
- attention entropy
- Confidence
- likelihood drift
- logit margin
- output KL divergence
- representation drift
- token-level KL
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
