Open-source LLM trustworthiness drifts significantly between checkpoints, study finds

A new study published on arXiv has audited the trustworthiness of open-source chat LLMs, specifically examining Yi, Qwen, Mistral, and Gemma across multiple generations. The research found that trustworthiness scores often do not accurately reflect changes between model checkpoints, with significant drift observed above a no-drift reference null. The authors propose that trust scores should be treated as dated artifacts specific to each checkpoint rather than being carried forward to subsequent versions without re-measurement, advocating for longitudinal model cards. AI

IMPACT Highlights the need for continuous evaluation of LLM trustworthiness as models evolve, impacting how benchmarks and model cards are interpreted.

RANK_REASON The cluster contains a research paper detailing findings on LLM trustworthiness. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Gemma
• Qwen

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →