New benchmark SEER-Bench tests LLM medical knowledge updating

Researchers have developed SEER-Bench, a new benchmark for evaluating how well large language models can update their medical knowledge. The benchmark uses oncology staging data and NCCN guidelines to test models under a matched training budget. Results indicate that the 'EMQ' supervision format is most effective for stable knowledge updating and retention, outperforming other formats like MSQ, FITB, and SAQ. A 4B model trained with EMQ supervision achieved competitive accuracy on temporally anchored oncology staging tasks. AI

IMPACT This research could lead to more reliable medical LLMs by improving how they are trained to incorporate new clinical information.

RANK_REASON The cluster is about a new academic paper introducing a benchmark and methodology for evaluating LLMs in a specific domain. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• 4B model
• arXiv
• HealthBench Professional
• large language models
• NCCN oncology guidelines
• SEER-Bench

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →