New training method boosts LLM visual sensitivity in machine translation

Researchers have developed a new training method called Metric-based Loss Weighting to enhance the visual sensitivity of Large Language Models (LLMs) in multimodal machine translation. This technique increases the loss function for tokens that can benefit from accompanying image information, identified using a Point-wise Cross-mutual Information (PCXMI) metric. When applied to Image-guided Machine Translation tasks, this method improved accuracy by up to 7 percentage points on the CoMMuTE dataset compared to standard fine-tuning, while preserving general translation performance. AI

IMPACT Enhances LLM capabilities in multimodal tasks, potentially improving translation accuracy and understanding of visual context.

RANK_REASON The cluster contains an academic paper detailing a new method for improving LLM performance on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• CoMMuTE
• Image-guided Machine Translation
• Large Language Models
• Metric-based Loss Weighting
• Point-wise Cross-mutual Information

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →