New Bayesian Penalty Reverses Attention Collapse in Language Models

Researchers have developed a new framework called Bayesian Repetition Penalty to address the issue of attention collapse in autoregressive language models. This pathology causes models to get stuck in repetitive loops. The framework works by comparing a token's observed frequency during generation against its corpus prior, using an adjacent-conditional probability construction. This method allows for a closed-form logit offset that can be applied as a frozen output-layer bias, effectively repairing collapsed models without altering their training pipelines. Experiments show this technique can significantly reduce repetition while maintaining generation quality. AI

IMPACT Offers a novel method to improve the reliability and quality of text generation from large language models by mitigating repetitive output.

RANK_REASON Academic paper detailing a new technical framework for language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• Attention Collapse
• autoregressive language models
• Bayesian Repetition Penalty
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →