Truth, Trust, and Trouble: Medical AI on the Edge
A new research paper introduces a framework for evaluating medical AI models on their truthfulness, usefulness, and safety. The study tested over 1,000 health questions across models like Mistral-7B, BioMistral-7B-DARE, and AlpaCare-13B. AlpaCare-13B demonstrated the highest accuracy and harmlessness, while BioMistral-7B-DARE showed improved safety through domain-specific tuning. The research also found that few-shot prompting enhanced accuracy, but all models struggled with helpfulness on complex medical queries. AI
IMPACT Establishes a benchmark for medical AI safety and accuracy, guiding future development and deployment in healthcare.