Apple researchers balance image captioning with new RL framework

Apple researchers have developed BalCapRL, a new framework for reinforcement learning-based image captioning using multimodal large language models. This approach aims to balance multiple caption quality dimensions, including correctness, reference coverage, and linguistic fluency, which are often compromised in existing methods. BalCapRL utilizes reward-decoupled normalization and length-conditional reward masking to optimize these objectives, showing significant improvements across various base models like LLaVA and Qwen. AI

IMPACT Introduces a novel approach to improve multimodal LLM image captioning by balancing multiple quality metrics, potentially enhancing downstream applications.

RANK_REASON The cluster contains a research paper from Apple Machine Learning Research detailing a new framework for image captioning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Apple Machine Learning Research →

• BalCapRL
• LLaVA-1.5-7B
• Qwen2.5-VL 3B
• Qwen2.5-VL 7B

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →