LLM repetition penalty flaw corrupts structured output, paper finds

A new paper identifies a critical flaw in the widely used multiplicative repetition penalty in LLM inference, which branches based on the sign of raw logits. This arbitrary branching point leads to the penalty being ill-defined, as re-centering logits should not affect the output at a penalty of 1.0, yet it alters a significant percentage of tokens. Furthermore, this penalty corrupts structured output, drastically reducing the rate of valid JSON schema outputs from 97% to 23%. The paper proposes and demonstrates that applying the penalty to normalized log-probabilities instead of raw logits resolves both issues, with HuggingFace already shipping this operator. AI

IMPACT Fixing this repetition penalty flaw could improve the reliability of structured data generation from LLMs.

RANK_REASON Academic paper detailing a technical flaw in LLM inference infrastructure. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• vLLM

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →