Researchers have identified a significant performance gap in multimodal large language models (MLLMs) when processing text presented as images compared to standard text tokens. This "modality gap" is primarily driven by the models' reduced reasoning effort when input is visual, leading to shorter, less computational outputs. A new self-distillation fine-tuning method, which pairs image inputs with the models' own reasoning traces from text mode, effectively closes this gap, improving accuracy and transferring gains to new benchmarks. AI
IMPACT Identifies a key limitation in MLLMs and proposes a method to improve their reasoning capabilities on visual text inputs.
RANK_REASON Academic paper detailing a new finding and method for multimodal LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →