New LLM data selection method balances mode-seeking and averaging

Researchers have introduced ICLHerd, a novel data selection framework designed to improve the training of large language models (LLMs). Unlike existing methods that focus solely on maximizing similarity to development sets, ICLHerd employs a von Mises-Fisher kernel to balance mode-seeking and mode-averaging. This approach aims to prevent a bias towards data that is too narrowly clustered around specific examples. The framework also utilizes in-context learning as an efficient proxy for hyperparameter tuning, demonstrating superior performance compared to current LLM data selection algorithms. AI

IMPACT This new data selection method could lead to more efficient and effective training of LLMs, potentially improving their performance on various downstream tasks.

RANK_REASON The cluster contains a research paper detailing a new method for LLM data selection. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CORE Recommender
• DagsHub
• Few-shot learning
• Gotit.pub
• Hugging Face
• ICLHerd
• LLMs
• ScienceCast
• von Mises-Fisher kernel

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →