Study finds AI thesis assessment struggles to match human priorities

A study surveyed 84 thesis supervisors across four academic disciplines to understand their prioritization of thesis assessment criteria. The research found significant differences between the supervisors' derived criterion weights and the default weights used by the AI assessment system RubiSCoT. When these supervisor-derived weights were integrated into RubiSCoT, the AI-generated evaluations showed only a marginal improvement in alignment with human assessments, indicating that criterion-weight calibration alone is insufficient to bridge the gap. AI

IMPACT This research highlights the challenges in aligning AI assessment tools with nuanced human evaluation criteria, suggesting further work is needed beyond simple weight calibration.

RANK_REASON The cluster contains an academic paper detailing an empirical study on AI-based thesis assessment. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• Litmaps
• RubiSCoT
• ScienceCast
• scite Smart Citations

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →