Researchers have developed SPY-Bench, a new benchmark designed to evaluate the ability of Large Vision-Language Models (LVLMs) to protect personalized privacy at inference time. The benchmark includes over 32,700 samples across various scenarios, revealing that even top-performing models like o4-mini struggle with compliance, achieving only about 24% accuracy. To address these limitations, the team implemented alignment methods, including a novel Noise Contrastive Alignment variant, which successfully reached 96.88% accuracy while preserving general performance, marking a step towards more controllable LVLMs. AI
IMPACT Establishes a new standard for evaluating and improving personalized privacy in LVLMs, crucial for ethical deployment.
RANK_REASON Academic paper introducing a new benchmark and method for evaluating LLM privacy. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
