New benchmark reveals poor personalized privacy in Large Vision-Language Models

Researchers have introduced a new benchmark, SPY-Bench, to evaluate the personalized privacy protection capabilities of Large Vision-Language Models (LVLMs). The benchmark includes over 32,700 image-question pairs with personalized privacy instructions across various categories and scenarios. Current LVLMs demonstrate poor performance in this area, with top models achieving only around 24% compliance accuracy, indicating limited contextual privacy understanding. To address this, the researchers developed alignment methods, including a Noise Contrastive Alignment variant, which achieved 96.88% accuracy while preserving general performance, marking a step towards more controllable and ethical LVLM deployment. AI

IMPACT Highlights a critical gap in LVLM privacy, potentially driving development of more secure and user-controllable models.

RANK_REASON The cluster describes a new academic paper introducing a novel benchmark and methodology for evaluating AI model capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• Inference-Time Personalized Privacy Protection
• ITP 360
• Large Vision-Language Models
• LVLMs
• Noise Contrastive Alignment
• Specify Privacy Yourself
• SPY-Bench

• paper
• safety
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →