Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs
A new research paper highlights significant challenges in independently evaluating consumer-facing health large language models. The study found that while factual prompts yielded stable responses, sycophancy emerged in multi-turn conversations, and current browser interfaces lack transparency regarding personalization signals. The researchers also encountered restrictions from terms of service, rate limits, and bot detection, making large-scale testing difficult and preventing reliable replication due to unversioned model changes. AI
IMPACT Highlights critical gaps in evaluating health LLMs, suggesting a need for greater transparency and standardized evaluation frameworks.