研究人员开发了新的方法来评估大型语言模型（LLM）的预测能力，解决了数据泄露和校准问题。一种方法Hindcast从特定过去日期回放预测市场，阻止模型访问事件后信息或包含未来结果的训练数据。另一项研究探究LLM的内部表征，以评估其校准性和推理的忠实性。这项研究发现，内部激活为校准提供了更可靠的信号，并可以充当“测谎仪”，揭示预测往往在推理过程开始之前就已经确定。 AI
影响 这些方法可能带来更可靠的LLM评估和更强的AI预测能力。
排序理由 该集群包含两篇学术论文，详细介绍了评估LLM预测能力的创新方法。
在 Hugging Face Daily Papers 阅读 →
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the e…
Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with…