English(EN)Self-Guided Test-Time Training for Long-Context LLMs
新研究探索自适应大语言模型评估和自我改进技术 · 追踪10个来源
作者PulseAugur 编辑部·[16 个来源]·
研究人员正在开发新的方法来评估和改进大语言模型（LLMs）。一种名为ATLAS的方法使用项目反应理论，显著减少了准确评估LLM所需的项目数量，需求减少高达90%。另一项研究将信号检测理论应用于衡量LLM的元认知效率，揭示置信信号和准确性并不总是对齐，并且在不同模型之间存在显著差异。此外，一个名为“Double Ratchet”的框架与LLM代理技能共同演进评估指标，即使在最初没有可靠指标的情况下也能实现自我改进。
AI
arXiv:2607.12790v1 Announce Type: new Abstract: Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists. In many real applications it does not. We make …
arXiv:2603.25112v2 Announce Type: replace-cross Abstract: Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate how much a model knows (Type-1 accuracy) with how well its confidence signal tracks that knowledge (Type-2 metacognitive …
arXiv cs.AI
TIER_1English(EN)·Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, Nitesh V. Chawla·
arXiv:2511.04689v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy ove…
Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists. In many real applications it does not. We make three claims. First, metrics can be \emph{evolve…
Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists. In many real applications it does not. We make three claims. First, metrics can be \emph{evolve…
arXiv cs.AI
TIER_1English(EN)·Tatiana Pelc, Gila Kamhi, Asaf Avrahamy, Adi Fledel-Alon·
arXiv:2607.11267v1 Announce Type: cross Abstract: In the rapidly evolving landscape of information retrieval systems, the ability to adapt and improve through user feedback is paramount. This study introduces a novel methodology for refining the performance of a primary Retrieval…
arXiv cs.AI
TIER_1English(EN)·Gabrielle Kaili-May Liu, Areeb Gani, Jacqueline Lu, Jordan Thomas, Mark Steyvers, Arman Cohan·
arXiv:2607.11881v1 Announce Type: cross Abstract: Metacognition is a foundational component of intelligence critical to effective learning, problem solving, decision-making, communication, and more. In recent years, it has become increasingly recognized as a cornerstone of capabl…
Metacognition is a foundational component of intelligence critical to effective learning, problem solving, decision-making, communication, and more. In recent years, it has become increasingly recognized as a cornerstone of capable, transparent AI systems. Yet while LLMs have mad…
Metacognition is a foundational component of intelligence critical to effective learning, problem solving, decision-making, communication, and more. In recent years, it has become increasingly recognized as a cornerstone of capable, transparent AI systems. Yet while LLMs have mad…
In the rapidly evolving landscape of information retrieval systems, the ability to adapt and improve through user feedback is paramount. This study introduces a novel methodology for refining the performance of a primary Retrieval Augmented Generation (RAG) system by strategicall…
In the rapidly evolving landscape of information retrieval systems, the ability to adapt and improve through user feedback is paramount. This study introduces a novel methodology for refining the performance of a primary Retrieval Augmented Generation (RAG) system by strategicall…
arXiv cs.AI
TIER_1English(EN)·Xinyu Zhu, Zhe Xu, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Kaushik Rangadurai, Hua Zhi, Frank Shyu, Sandeep Pandey, Luke Simon, Yu Meng, Xi Liu·
arXiv:2607.09415v1 Announce Type: cross Abstract: Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often deg…
Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to id…
Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to id…
X — Omar Sanseviero (HF research)
TIER_1English(EN)·omarsar0·
Highly-recommended overview of metacognition in LLMs.
(bookmark it)
Interesting behaviors in LLMs like confidence calibration, self-verification, knowing when to stop, and knowing what you do not know have mostly been studied in isolation.
This survey argues they are facets of…
<p><strong>What I Learned Using LLMs for Step-by-Step Explanations</strong></p> <p>LLMs are good at producing explanations, but a generated explanation is not automatically a useful one.</p> <p>That was one of the clearest lessons from building a small AI study workflow. The app …
