New framework evaluates MLLMs for thermal grounding in infrared images

A new evaluation framework for multimodal large language models (MLLMs) applied to infrared images has been developed, which goes beyond simple answer accuracy to assess the grounding of explanations in thermal evidence. This framework, utilizing a Dual-LLM Consensus Judge, revealed that even correct answers can be based on weak or visible-light evidence, and that withholding infrared imagery degrades thermal grounding, particularly in more capable models. The researchers also introduced Thermal-Grounded Feedback (TGF), a training-free method to revise explanations and improve their thermal grounding without altering the selected answer, suggesting a need for MLLMs to prioritize thermally grounded explanations for reliable infrared scene understanding. AI

IMPACT Highlights the need for more robust evaluation of MLLMs, especially in specialized domains like infrared imaging, to ensure reliable and trustworthy outputs.

RANK_REASON Academic paper introducing a new evaluation framework and method for multimodal large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• Dual-LLM Consensus Judge
• Infrared images of the transiting disk in the epsilon Aurigae system
• MLLMs
• Thermal-Grounded Feedback

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →