New AI model Composer uses proxy-tokens for improved visual reasoning and interpretability

Researchers have developed a new multimodal large language model (MLLM) called Composer, designed to improve the interpretability and trustworthiness of AI systems. Composer utilizes learned proxy-tokens to explicitly link textual rationales with visual grounding information, addressing the semantic-spatial gap found in current models. This novel mechanism allows the model to treat visual regions as addressable sets, enhancing the accuracy of visual grounding by 9.0 points while maintaining performance in final answer accuracy. A new dataset, ComposerGCoT, was created to rigorously evaluate this grounding mechanism, demonstrating that discrete proxy-tokens are more effective than traditional textual coordinates for capturing spatial semantics. AI

IMPACT Introduces a novel approach to MLLM interpretability, potentially increasing trust and reliability in critical AI applications.

RANK_REASON The cluster contains an academic paper detailing a new model and dataset. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Composer
• ComposerGCoT
• Multimodal Large Language Models

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →