New Omni-Persona benchmark evaluates omnimodal LLM personalization

Researchers have introduced Omni-Persona, a novel benchmark designed to systematically evaluate and improve omnimodal personalization in large language models. This benchmark addresses limitations in existing research by covering text, image, and audio modalities, and by introducing a new metric, Calibrated Accuracy (Cal), to assess both correct grounding and appropriate abstention, even in absent-persona scenarios. Experiments revealed a notable audio-vs-visual grounding gap in open-weight models, with RLVR partially mitigating this issue, while also highlighting that recall and model scale are insufficient diagnostics for personalization quality, underscoring the need for calibration-focused evaluation. AI

IMPACT Establishes a new standard for evaluating LLM personalization across multiple modalities, guiding future research in model training and reward design.

RANK_REASON The cluster contains a research paper detailing a new benchmark and evaluation metric for LLM personalization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Calibrated Accuracy
• Omni-Persona
• Persona Modality Graph
• RLVR
• supervised fine-tuning
• University of California, Berkeley
• Yeongtak Oh

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →