LLMs outperform decision models on medical QA, but cost and speed vary

A new study published on arXiv benchmarks traditional decision models against generative large language models (LLMs) using medical multiple-choice questions from the MedQA dataset. The research found that while LLM configurations achieved higher accuracy, ranging from 78.32% to 96.86%, compared to decision models (67.71% to 89.08%), specific decision models like Mercury Decide offered significantly lower costs per query. Clef Omni demonstrated the shortest request duration among the tested configurations. The study also highlighted that decision models' probabilities could distinguish correct answers, but their error rates at high confidence levels and probability repeatability require careful consideration alongside accuracy and resource needs when selecting models for medical workflows. AI

IMPACT Highlights trade-offs between LLM accuracy and decision model cost/speed for specialized tasks like medical QA.

RANK_REASON The cluster contains a research paper published on arXiv detailing a benchmark comparison of AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• Litmaps
• MedQA
• Mercury Decide
• ScienceCast
• scite Smart Citations

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →