VLMs struggle to reliably assess sidewalk accessibility, study finds

Researchers have investigated the capability of vision-language models (VLMs) to assess sidewalk accessibility attributes from pedestrian-level imagery. Using sampling-based conformal prediction, they evaluated four VLMs on 514 images from Seoul, South Korea, comparing model outputs to field-measured ground truth. While conformal calibration achieved nominal coverage, the informativeness varied significantly by attribute, with effective width being the most precise. The study found that no quantitative attribute reached the precision needed for general compliance assessment, highlighting the importance of calibrated uncertainty estimation over raw response self-consistency. AI

IMPACT This research highlights limitations in current VLMs for real-world spatial understanding and suggests methods for more reliable uncertainty quantification in AI systems.

RANK_REASON Academic paper detailing a novel application of conformal prediction to VLMs for accessibility assessment. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• conformal prediction
• Hugging Face
• Seoul, South Korea
• sidewalk accessibility
• Vision--Language Models

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →