Lightweight probabilistic method matches LLM performance on clinical diagnosis

A new research paper introduces a lightweight probabilistic method called Frequency-Based Probabilistic Ranker (FBPR) that can match the performance of large language models (LLMs) on multiple-choice clinical diagnosis questions from the MedQA dataset. FBPR utilizes a smoothed Naive Bayes classifier trained on concept-diagnosis co-occurrence statistics from corpora like Dolma and RedPajama. The study found that FBPR and direct LLM inference have complementary strengths, and combining them through a hybrid approach significantly improves accuracy over either method alone. This suggests that explicit probabilistic reasoning can account for a substantial portion of LLM performance on such diagnostic tasks. AI

IMPACT Demonstrates that explicit probabilistic methods can achieve comparable performance to LLMs on specific diagnostic tasks, suggesting potential for hybrid approaches.

RANK_REASON Research paper introducing a new method and benchmark results. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Dolma
• Frequency-Based Probabilistic Ranker
• LLaMA
• MedQA
• naive Bayes classifier
• OLMo
• RedPajama

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →