TANDEM method optimizes LLM training data mixtures with twin networks

Researchers have introduced TANDEM, a novel method for optimizing the mixture ratios of training data used for large language models. TANDEM employs a bi-level optimization approach, simplified into a single-level penalized form solved by twin networks. This system measures data efficacy by comparing a primary model with a dynamically updated reference model, up-weighting domains that show greater benefit from additional data. The method offers theoretical guarantees and has demonstrated effectiveness across various scenarios, including data-restricted settings and supervised fine-tuning. AI

IMPACT Optimizes LLM training data mixtures, potentially improving model performance and efficiency.

RANK_REASON The cluster contains a research paper detailing a new method for optimizing LLM training data. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• LLMs

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →