研究人员开发了两种评估大型语言模型（LLM）的新方法。SelfReflect 评估 LLM 报告的自身不确定性是否与其真实的响应变异性一致，发现通常不一致，除非模型经过自身答案示例的专门训练。另一方面，KGLens 将知识图谱转化为测试问题，以查明模型的知识弱点并绘制其在不同知识领域的可靠性。 AI
影响 新的评估技术可以通过更好地识别事实不准确性和不确定性来提高 LLM 的可靠性和安全性。
排序理由 该集群描述了研究论文中提出的 LLM 的新颖评估方法。
在 Mastodon — fosstodon.org 阅读 →
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
SelfReflect measures whether an LLM's text summary of its uncertainty matches its actual answer distribution. Across 20 modern models: it doesn't, unless the model sees samples of its own answers first. The negative result does more work than the metric itself. Fits a growing lin…
KGLens turns a knowledge graph into test questions and uses Thompson sampling to zero in on a model's weakest facts. The interesting bit is the output shape: a per-relation map of where the model is and isn't reliable, against a graph matched to your deployment. Sampling trick sh…