AI models alter student quotes in feedback, study finds

A study by MV2 of Munim, Inc. investigated whether AI models accurately quote student text in their feedback. Across six different large language models, including Llama-3.1:8b, Mistral-7B, Mistral-24B, Qwen2.5-70B, Gemini 2 Flash, and LearnLM, the research found that approximately 1.18% of quotes in AI-generated feedback altered the student's original words. Shorter quotes were more likely to be altered than longer ones, and the study revealed that model size was not a reliable indicator of quoting accuracy, with one instruction-tuned model being highly faithful and a mid-size general model performing poorly. AI

IMPACT Highlights the need for careful evaluation of AI feedback tools, as accuracy varies significantly between models.

RANK_REASON Research paper analyzing AI model behavior on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Almousa
• FeedType
• Gemini 2 Flash
• IELTS
• LearnLM
• Llama-3.1:8b
• Mistral-24B
• Mistral-7B
• Munim, Inc.
• Qwen2.5-70B

• product
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →