InfiGFusion fuses LLMs using graph-on-logits distillation

Researchers have developed InfiGFusion, a novel framework for merging heterogeneous open-source large language models. This method uses a Graph-on-Logits Distillation (GLD) loss to model semantic dependencies between tokens, which previous methods overlooked. InfiGFusion significantly improves fusion quality and stability, outperforming state-of-the-art baselines on 11 benchmarks, particularly in complex reasoning tasks. AI

IMPACT Introduces a new method for improving the performance of fused LLMs, especially in complex reasoning tasks.

RANK_REASON The cluster contains an academic paper detailing a new method for model fusion. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Graph-on-Logits Distillation
• InfiGFusion
• large language models
• Yuanyi Wang

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →