FaithEyes framework improves VLM tool use faithfulness

Researchers have introduced FaithEyes, a novel multi-agent framework designed to enhance the faithfulness of tool use in agentic vision-language models (VLMs). These models often struggle with tool calls that are irrelevant to the task, leading to inaccurate reasoning despite correct final answers. FaithEyes addresses this by having the VLM itself judge the usefulness of each process image generated by tool calls. This judgment is integrated into the reasoning context and used to scale tool rewards, thereby mitigating reward hacking. The framework ensures train-test consistency by employing the model as a subagent for self-judging, and has demonstrated competitive accuracy on visual perception and reasoning benchmarks while significantly improving tool faithfulness. AI

IMPACT Enhances reliability and interpretability of multimodal reasoning in VLMs by ensuring tool use is relevant to the task.

RANK_REASON The cluster contains a research paper detailing a new framework for improving AI model performance. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• CORE Recommender
• DagsHub
• FaithEyes
• Gotit.pub
• Hugging Face
• Litmaps
• ScienceCast
• scite Smart Citations

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →