From Outliers to Errors: Auditing Pali-to-English LLM Translations with Multi-Reference Adjudication
Researchers have developed a new method to audit Large Language Model (LLM) translations of Pali to English, addressing the challenge of single-score metrics conflating valid variations with errors. The study utilized multiple established human translations as a reference envelope and employed embedding drift to identify potential issues in LLM outputs. This approach allowed for a more nuanced evaluation, distinguishing between genuine errors and acceptable translation differences, particularly for classical languages. AI
IMPACT Introduces a novel audit design for classical language translation, potentially improving LLM evaluation standards.