New benchmark TreeProbe assesses LLM bias in Tibetan medicine

Researchers have developed TreeProbe, a new benchmark designed to evaluate cultural bias in large language models concerning traditional Tibetan medicine. This benchmark, organized around the native "Tree of Medicine" framework, contains over 4,700 expert-adjudicated items covering diseases and subtasks. Experiments using TreeProbe revealed that current LLMs struggle with Tibetan medical contexts, showing a tendency to drift towards either biomedical or traditional Chinese medicine reasoning based on their training data. AI

IMPACT This benchmark could lead to more culturally sensitive and equitable AI systems in healthcare, particularly for underrepresented medical traditions.

RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• DagsHub
• Hugging Face
• LLMs
• traditional Chinese medicine
• traditional Tibetan medicine
• TreeProbe

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →