New framework guides LLM layer updates for efficient pre-training

Researchers have developed LayerTracer, a new framework to guide the selective updating of large language model layers during continued pre-training. This method analyzes layer representation evolution and sensitivity to identify which layers are critical for task execution and stability. Experiments show that freezing deep layers while training shallow ones leads to better performance on benchmarks like C-Eval and CMMLU compared to full parameter fine-tuning or the reverse strategy. AI

IMPACT Provides a low-cost, interpretable method for optimizing LLM continued pre-training, benefiting resource-constrained teams.

RANK_REASON The cluster contains an academic paper detailing a new framework and experimental results for continued pre-training of LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• CMMLU
• Large Language Models
• LayerTracer
• Yuhang Wu

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →