Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity
A new research paper compares the performance of large language models (LLMs) against fine-tuned RoBERTa models for extracting complex circumstances from death investigation narratives. The study introduces a "Complexity Score" algorithm to determine optimal prompting strategies, finding that LLMs excel at low-prevalence circumstances where fine-tuned models lack sufficient training data. The research demonstrates consistent performance patterns across frontier LLMs like GPT-5.2, Gemini 2.5 Pro, and Llama-3 70B, suggesting a hybrid architecture where LLMs handle rare cases and fine-tuned models manage common ones. AI
IMPACT Suggests a hybrid LLM architecture for specialized data extraction tasks, potentially improving efficiency in fields like public health.