DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer
Researchers have developed DuDi, a novel dual-signal distillation framework designed to enhance the multilingual capabilities of small language models (SLMs). This method combines sequence-level and token-level signals, incorporating a cross-lingual verbalizer to refine teacher feedback. Experiments demonstrate that DuDi significantly improves performance on Southeast Asian languages, outperforming existing distillation techniques across various model scales and families. AI
IMPACT Enhances multilingual capabilities of small language models, potentially improving accessibility and performance for under-resourced languages.