New CIGA method improves LLM data selection by focusing on content

Researchers have developed a new method called Content-Invariant Gradient Attribution (CIGA) to improve the selection of training data for large language models (LLMs). Existing gradient-based methods are susceptible to minor, semantically irrelevant changes in data, leading to inconsistent attribution scores. CIGA addresses this by identifying stable gradient layers and fusing them with semantic representations, ensuring that attribution is based on content rather than superficial formatting. Experiments demonstrate that CIGA provides more stable data rankings, maintains downstream capabilities, and enhances overall performance while reducing computational costs. AI

IMPACT Enhances LLM training efficiency and performance by improving data selection robustness.

RANK_REASON This is a research paper detailing a new method for LLM data selection. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• CIGA
• Connected Papers
• Content-Invariant Gradient Attribution
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv Recommender
• large language models
• Litmaps
• LLMs
• ScienceCast
• scite Smart Citations

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →