InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion
Researchers have developed InfiGFusion, a novel framework for merging heterogeneous open-source large language models. This method uses a Graph-on-Logits Distillation (GLD) loss to model semantic dependencies between tokens, which previous methods overlooked. InfiGFusion significantly improves fusion quality and stability, outperforming state-of-the-art baselines on 11 benchmarks, particularly in complex reasoning tasks. AI
IMPACT Introduces a new method for improving the performance of fused LLMs, especially in complex reasoning tasks.