A test of six AI assistants on everyday questions about real places revealed significant variability in their responses. The PlaceBench test asked 20 questions three times each, finding that the AI model Instinct changed its grade on 15 of those questions, while another model, Dots, was consistently accurate. The study also calculated the probability of receiving correct answers across multiple attempts, with Dots showing the highest reliability. AI
IMPACT Highlights the need for developers to account for AI response variability when building applications on top of these models.
RANK_REASON The item details a methodology and results from a test of AI assistant consistency. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →