A new benchmark, AusClinBench, has been developed to evaluate the performance of large language models (LLMs) against current Australian clinical policies. The benchmark consists of 142 questions across five domains, including the Pharmaceutical Benefits Scheme (PBS) and Medicare Benefits Schedule (MBS) billing, with answers scored by exact match. Testing revealed that blind accuracy ranged from 51.5% to 92.7%, with the highest-scoring models being indistinguishable. A significant finding was that models often provided incorrect answers when not given the governing clause, highlighting a failure to recall current rules. AI
IMPACT Establishes a new standard for evaluating LLM accuracy in specialized clinical domains, potentially influencing the development and deployment of AI in healthcare.
RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating LLMs on specific domain knowledge. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- AusClinBench
- Australian clinical policy
- Australian Health Practitioner Regulation Agency
- Hugging Face
- large language models
- PBS
- Pharmaceutical Benefits Scheme
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
