New methods combat text degeneration in pruned large language models

Researchers have developed two new methods, FOCUS and RePAIR, to address text degeneration issues in pruned large language models (LLMs). Pruning, a technique to compress LLMs, can inadvertently increase problems like repetition loops without significantly impacting perplexity or task accuracy. These new methods analyze degeneration at the token level, identifying loop entry and persistence as key factors. FOCUS reweights distillation to high-confidence areas, while RePAIR uses specific continuation pairs to encourage plausible alternatives and prevent early commitment to repetitive outputs. Experiments demonstrate that both FOCUS and RePAIR effectively reduce repetition and enhance generation quality in open-ended and instruction-based tasks. AI

IMPACT These methods could improve the quality and reliability of smaller, more efficient language models for various applications.

RANK_REASON The cluster contains a research paper detailing new methods for improving pruned large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• FOCUS
• Hugging Face
• large language models
• RePAIR

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →