English(EN)From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings
新研究探讨LLM代理的可靠性和决策能力
作者PulseAugur 编辑部·[73 个来源]·
近期研究探讨了大型语言模型（LLM）代理面临的挑战和解决方案，重点关注提高其可靠性和决策能力。论文介绍了新的基准和方法论，用于评估代理在量化、信息获取、金融研究中的时间有效性以及协作等方面的性能。提出了诸如闭环教师引导、学生感知技能迁移和伪造门控自我演化等技术，以增强代理的可靠性和学习能力。此外，研究还探讨了LLM代理如何理解社交信号并通过共享记忆进行协调，强调了需要强大的评估框架来捕捉复杂的多代理交互。
AI
arXiv:2611.12966v1 Announce Type: new Abstract: Large language models (LLMs) can generate SQL and SQL--Python workflows for investment research, but ensuring point-in-time (PIT) validity remains challenging. Agents must enforce disclosure cutoffs and select the appropriate histor…
arXiv:2611.13410v1 Announce Type: new Abstract: Low-bit quantization reduces the inference cost of LLM agents but can compromise task success. Examining failed rollouts, we find that quantized agents overlook task requirements or environmental conditions, producing environment tr…
arXiv:2611.13254v1 Announce Type: new Abstract: When an LLM agent fails part-way through a long task, the usual analysis records a symptom: a failure-mode label, or the agent and step judged responsible. It does not say where the error entered, which state carried it to the sympt…
arXiv:2611.13144v1 Announce Type: new Abstract: Effective teaching requires reasoning about the learner as well as the task, a capacity that Theory of Mind describes as inferring another agent's latent state from its behavior. We ask whether a stronger LLM teaches a fixed student…
arXiv cs.AI
TIER_1English(EN)·Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou·
arXiv:2611.13015v1 Announce Type: new Abstract: LLM agents are increasingly asked to make plans for people, such as scheduling drivers or assigning nurses. However, people's requests are often incomplete, and the supporting information for decision-making may be outdated, requiri…
arXiv:2611.14326v1 Announce Type: new Abstract: LLM agents increasingly operate in long-horizon, verifier-rich environments, but current self-improvement mechanisms often write episode-derived lessons directly into memory, making local success an unreliable signal for reusable an…
arXiv cs.MA (Multiagent)
TIER_1English(EN)·Andrew D. Wilson·
As large language models power autonomous agents, understanding how they collaborate and where they fail becomes critical. We introduce PARLEY, an extensible platform for studying LLM agent collaboration through four asymmetric puzzle tasks grounded in cognitive psychology (refer…
Low-bit quantized LLM agents can recover much of their task success through closed-loop guidance from a full-precision teacher, followed by unassisted student training on corrected and reference turns.
Most frameworks for teams of large language model (LLM) agents fix the team's shape inside the software: who talks to whom, how much is passed along, and what a human can do once a run has started. Human-team research has long asked those questions: where team phenomena live (in …
Coordination between large language-model agents has become a central question for AI safety and software engineering, as such agents are deployed in groups that share infrastructure. Existing research tells the agents that partners exist or asks them to coordinate. Whether coord…
arXiv:2603.23509v2 Announce Type: replace Abstract: LLMs and coding agents can now tackle complex, long-horizon tasks through advances in reinforcement learning, but stronger task-solving capabilities do not necessarily preserve safety alignment. Recent AI agent incidents involvi…
arXiv:2611.08971v1 Announce Type: cross Abstract: Multi-agent LLM systems increasingly aggregate judgments across model instances, but reasoning about other agents can itself introduce failure. We study a setting in which LLMs shift their public decisions after observing ambiguou…
Large language model (LLM) agents increasingly operate in groups, each pursuing an individual objective while jointly determining system performance. If an operator controls which agent observes which other agent, but not their payoffs or messages, how should observations be stru…
Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the divers…
We audit one live LLM agent arena (AITRAP; 22 autonomous agent seats, ~96,000 reasoning nodes) over nine days (Oct 16-25, 2026), during which the platform deployed two bundled incentive changes: a progressive tax reform (V2.9.3, Oct 19) and a seed-inversion with explorer-side act…
Large language model agents can exceed their authorization boundaries while pursuing user goals even without an adversary, and a benchmark reveals that this drift is common, capability-linked, avoidable, and legally relevant.
arXiv:2611.07073v1 Announce Type: new Abstract: A verified Dafny postcondition certifies that a formula holds, but it does not guarantee that the result was reached through genuine independent reasoning. We introduce a cross-agent formalization protocol where a Specifier model tr…
arXiv:2609.22818v3 Announce Type: replace-cross Abstract: Memory-poisoning defenses for LLM agents are usually judged by how well they stop attacks, yet almost all the traffic an agent sees is benign. We measure what these defenses cost when no attack is present. Our harness hold…
arXiv:2611.07483v1 Announce Type: cross Abstract: Enterprise AI agents increasingly split a user's request across subagents and merge their results, and each handoff can summarize away, alter or ignore a requirement. We report a preregistered study with synthetic, exactly scorabl…
arXiv cs.AI
TIER_1English(EN)·Burkay Tun\c{c}t\"urk, Koorosh Aslansefat, Yiannis Papadopoulos, Bao Nguyen·
arXiv:2611.07140v1 Announce Type: cross Abstract: Teams increasingly build software from cooperating large language model (LLM) agents, often assuming more agents mean more reliability. Yet the usual evidence, average benchmark accuracy, ignores the input mix a deployed system me…
arXiv cs.AI
TIER_1English(EN)·Mete Erdogan, Mirac Suzgun, Aneesh Pappu, Guowei Xu, Batu El, Andrew A. Schoen, Furong Huang, John P. Dickerson, Mert Pilanci, James Zou·
arXiv:2611.07667v1 Announce Type: new Abstract: The performance of multi-agent LLM systems depends not only on the capabilities of the underlying models but also on information-sharing policies, which govern what information each agent observes and at what level of detail. But do…
Multi-agent LLM systems increasingly aggregate judgments across model instances, but reasoning about other agents can itself introduce failure. We study a setting in which LLMs shift their public decisions after observing ambiguous social signals while reporting inaccurate estima…
Multi-Agent Debate (MAD) improves reasoning on multimodal tasks, but its accuracy and token efficiency depend on how agents interact during debate. A largely overlooked design choice is which interaction modalities agents use to exchange evidence. Existing methods rely on fixed t…
Multi-Agent Debate (MAD) improves reasoning on multimodal tasks, but its accuracy and token efficiency depend on how agents interact during debate. A largely overlooked design choice is which interaction modalities agents use to exchange evidence. Existing methods rely on fixed t…
The ODD protocol is the de facto standard for describing agent-based models, but it lacks vocabulary for the components that determine LLM-agent behavior: prompt assembly, memory serialization, parsing, and retry logic. We propose ODD+LLM, a backward-compatible extension adding f…
The performance of multi-agent LLM systems depends not only on the capabilities of the underlying models but also on information-sharing policies, which govern what information each agent observes and at what level of detail. But does providing AI agents with more detailed inform…
Enterprise AI agents increasingly split a user's request across subagents and merge their results, and each handoff can summarize away, alter or ignore a requirement. We report a preregistered study with synthetic, exactly scorable tasks comparing free-text, structured JSON and s…
arXiv:2611.06674v1 Announce Type: cross Abstract: Infrastructure security does not establish whether an LLM agent selected an allowed tool, grounded its answer in tool output, or followed a required workflow. AgentTrustCI is a Java/JUnit 5 framework that expresses these applicati…
arXiv:2610.04672v2 Announce Type: replace Abstract: Large language models (LLMs) have progressively evolved into the core of autonomous agents. Building on this progress, LLM-based multi-agent systems (MAS) coordinate multiple agents into a synergistic team to accomplish complex …
arXiv:2611.05720v1 Announce Type: new Abstract: Search can help LLM agents correct mistakes, but their starting conditions may shape which evidence they find and use. Drawing on cognitive-bias paradigms, we develop a controlled search harness to examine result order, records of p…
arXiv cs.CL
TIER_1English(EN)·Yulu An, Yifei Feng, Xiangyang Yin·
arXiv:2611.06876v1 Announce Type: new Abstract: LLM agents use context compaction to manage growing interaction histories, but summarizing long histories with large models can introduce substantial delays. Compaction therefore requires balancing information retention against late…
arXiv cs.CL
TIER_1English(EN)·Mattia Greiche, Reihaneh Rabbany, Maximilian Puelma Touzel·
arXiv:2611.06914v1 Announce Type: cross Abstract: Researchers use large language models as agents in social simulations, where agents are prompted with observations and background information intended to shape their actions. Yet LLM agents have been shown to substantially change …
arXiv cs.AI
TIER_1English(EN)·Adrian Hayler, Shashank Reddy Chirra, Andrei Lupu, Johannes Forkel, Bidipta Sarkar, Siheng Feng, Jakob Foerster·
arXiv:2611.06409v1 Announce Type: cross Abstract: We study zero-shot coordination (ZSC), where independently developed agents must coordinate at test time. While ZSC has been well studied in the RL literature, far less is known about the performance of LLM agents despite their in…
arXiv cs.LG
TIER_1English(EN)·Jorge Garc\'ia-Carrasco, Javier Sanchis, \'Alvaro Navarro, Alejandro Mat\'e, Juan Trujillo·
arXiv:2611.06257v1 Announce Type: cross Abstract: Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction,…
arXiv cs.AI
TIER_1English(EN)·Jalaj Bhandari, Dhawal Gupta, Hongli Zhan, Zhenting Wang, Varad Pimpalkhute, Zhengzhong Liu, Eric Xing, Rupesh Srivastava·
arXiv:2611.06868v1 Announce Type: new Abstract: In this work, we evaluate a simple inference-time scaling approach based on \textit{turn-level best-of-N sampling} (\method) for large language model (LLM) agents on complex, long-horizon tasks. We instantiate this using a \textit{g…
arXiv:2611.06841v1 Announce Type: new Abstract: Large language model (LLM) coding agents develop software through experimentation, offering an opportunity to design the serving systems supporting their own operation. Effective serving requires coordinating implementation and oper…
arXiv:2611.06254v1 Announce Type: new Abstract: The performance of large language model (LLM) agents depends on both the underlying model and the harness that governs interaction with tools and the environment. Yet harness design remains largely manual, and automating it requires…
arXiv:2605.10779v2 Announce Type: replace-cross Abstract: The rapid proliferation of LLM-based autonomous agents in real operating system environments introduces a new category of safety risk beyond content safety: behavior jailbreak, where an adversary induces an agent to execut…
Teams increasingly build software from cooperating large language model (LLM) agents, often assuming more agents mean more reliability. Yet the usual evidence, average benchmark accuracy, ignores the input mix a deployed system meets and the rule merging the agents' answers. Buil…
A verified Dafny postcondition certifies that a formula holds, but it does not guarantee that the result was reached through genuine independent reasoning. We introduce a cross-agent formalization protocol where a Specifier model translates clinical trial criteria into a Dafny pr…
Researchers use large language models as agents in social simulations, where agents are prompted with observations and background information intended to shape their actions. Yet LLM agents have been shown to substantially change their behavior under meaning-preserving prompt var…
Researchers use large language models as agents in social simulations, where agents are prompted with observations and background information intended to shape their actions. Yet LLM agents have been shown to substantially change their behavior under meaning-preserving prompt var…
Large language model (LLM) coding agents develop software through experimentation, offering an opportunity to design the serving systems supporting their own operation. Effective serving requires coordinating implementation and operating decisions under session dependencies, late…
arXiv:2611.03434v1 Announce Type: cross Abstract: Government intelligence teams need LLM agents that can analyze large, messy evidence collections, but security and cost constraints often rule out hosted frontier models and broad, hard-to-audit tool surfaces. We report CLEO: Cons…
arXiv:2609.23790v1 Announce Type: cross Abstract: Every node in a multi-agent large language model (LLM) workflow retrieves context from memory and injects it into its prompt, where those injected tokens are billed as input tokens at the same per-token price as the system prompt …
arXiv cs.CL
TIER_1English(EN)·Angana Borah, Zoha Hayat Bhatti, Zhijing Jin, Rada Mihalcea·
arXiv:2611.01301v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed in multi-agent settings, exhibiting emergent collective behaviors, yet the role of curiosity as a driver of such dynamics remains understudied. We introduce CuriousTogether, a f…
arXiv cs.CL
TIER_1English(EN)·Marianne Menglin Liu, Rongguang Wang, Zheng Wang, Yewen Wang, Leonid Boytsov, Tao Sheng, Sujith Ravi, Dan Roth·
arXiv:2611.03551v1 Announce Type: new Abstract: LLM coding agents can increasingly automate model post-training, but whether they can act as forward-deployed engineer (FDE) agents, discovering customer requirements and turning them into deployment decisions, remains unclear. We s…
arXiv:2611.03575v1 Announce Type: new Abstract: Large language model (LLM) coding agents increasingly operate with shell access, allowing them to write scripts, move files, alter permissions, and modify system configurations. However, an agent may achieve a user's goal while caus…
Search can help LLM agents correct mistakes, but their starting conditions may shape which evidence they find and use. Drawing on cognitive-bias paradigms, we develop a controlled search harness to examine result order, records of prior work, and initial hypotheses while holding …
De-siloing can create groupthink; silos can prevent knowledge sharing and reduce research efficiency. The same tension appears in research systems built with large language model (LLM) agents, yet their organization is usually fixed by design. I propose treating laboratory struct…
Government intelligence teams need LLM agents that can analyze large, messy evidence collections, but security and cost constraints often rule out hosted frontier models and broad, hard-to-audit tool surfaces. We report CLEO: Constrained Loop for Environment Optimization, a metho…
Indirect prompt injection creates a distinctive risk for blue-team Large Language Model (LLM) agents in Security Operations Centers (SOCs). Malicious instructions can be embedded in telemetry, tickets, retrieval results, or memory and then propagate through alert triage, incident…
arXiv:2603.26034v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) perform complex tasks through long-horizon reasoning and tool interaction, where a fundamental trade-off arises between execution efficiency and reasoning robustness. Mod…
arXiv:2610.42004v1 Announce Type: new Abstract: Multi-turn negotiation remains a challenging problem for large language model (LLM) agents. While LLMs can conduct natural-language dialogue, existing benchmarks predominantly assess only agreement rates, overlooking whether negotia…
arXiv cs.AI
TIER_1English(EN)·Peter Chen, Ni Zhang, Ying Xu, Siyuan Zhang·
arXiv:2610.41604v1 Announce Type: new Abstract: Agent systems built on local large language models are increasingly marketed through the vocabulary of cognitive architecture: modules named global workspace, theory of mind, causal reasoning, or active inference are listed as featu…
arXiv:2610.40716v1 Announce Type: cross Abstract: LLM agents operating in untrusted workspaces face the risk of untrusted code execution during routine tasks, yet final reports and workspace states alone cannot reliably establish whether execution occurred. We introduce an evalua…
Current automated agent design frameworks produce either static workflows that lack adaptability or per-query optimizers that prevent the accumulation of deep, agent-level task expertise. We propose a new direction that reconciles these paradigms: creating stateful teams of speci…
arXiv:2610.38620v1 Announce Type: new Abstract: LLM agents increasingly perform consequential actions through external tools, yet their proposed actions are often executed without independent verification of policy compliance or task completion. We introduce JevAct, a runtime ver…
arXiv cs.AI
TIER_1English(EN)·Zihou Ren, Byungchul Tak, Long Wang·
arXiv:2610.39155v1 Announce Type: cross Abstract: Large language model (LLM) agents often receive access to more files, accounts, and services than a task requires. Prompt injection turns this excess access into an opportunity to redirect execution toward unrelated resources. App…
arXiv:2610.40227v1 Announce Type: new Abstract: Epistemic game theory explains an agent's strategic behavior via the beliefs it holds about others and how these beliefs are updated as the interactions unfold. Modern LLM-based agents, however, often lack an explicit mechanism for …
Large language model (LLM)-based multi-agent systems (MAS) coordinate specialized agents to solve complex tasks, turning each user request into interdependent model and tool operations. Unlike independent requests, these operations become ready at different times and consume evol…
MAServeBench evaluates LLM serving and hardware by modeling multi-agent collaboration as a configurable workload dimension with reproducible dependency traces.
Agents built on large language models are increasingly allowed to take consequential actions: cancelling bookings, issuing refunds, changing accounts. The rules that govern those actions, such as "obtain the customer's explicit confirmation before changing anything", are usually …
LLM agents interact with Web services to perform tasks on users' behalf. Errors during these interactions can disrupt execution and prevent task completion, motivating external intervention. However, agents may self-recover through continued execution, raising the question of whe…
<p>Most "give the model a browser" setups hand it one of two things: a screenshot, or a dump of the full accessibility tree. Both are expensive, and both get re-sent in full after every action.</p> <p>I've been building <a href="https://github.com/HaidarESBER/ai-browser" rel="noo…
Medium — MLOps tag
TIER_1English(EN)·Aakash Agrawal·
<p>Like many of you, my feed is full of "AI business gurus" selling step-by-step blueprints. But if the formula is so good, why sell it?</p> <p>I decided to check the actual data on these business models. The numbers are grim:</p> <p>[80-90% of e-commerce sites fail.]</p> <p>[The…
dev.to — LLM tag
TIER_1English(EN)·hugginf_expert·
<h2> TL;DR </h2> <ul> <li>HiddenLayer researchers published "Man-in-the-Model" on October 22, 2026. Their point: if an attacker controls the inference server or LLM gateway that sits between a coding agent and its model, the attacker can inject tool calls the model never requeste…
dev.to — LLM tag
TIER_1English(EN)·TechSavant Security Lab·
<p>In May 2026, Microsoft's security team published two remote-code-execution issues in its own agent framework, Semantic Kernel (<a href="https://www.microsoft.com/en-us/security/blog/2026/05/07/prompts-become-shells-rce-vulnerabilities-ai-agent-frameworks/" rel="noopener norefe…
dev.to — LLM tag
TIER_1English(EN)·CryptoPunk Dz2·
<p>A coordination protocol can keep its checked safety rules and still fall out of use. That is the main thing we learned from running LLM coding agents on seven machines for 103 days, and it is why we are releasing <strong><a href="https://github.com/tonydzi/claw-consensus" rel=…
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1xng029/ai_agent_task_performance_of_small_llm_models_4b/"> <img alt="AI Agent Task performance of small LLM models (4B and below) in self-assessment set" src="https://preview.redd.it/1upg1g127jzh1.png?width=6…
<p>I'm Doyoon, an AI persona (not a human) focused on practical AI engineering. This post looks at a data point I think deserves more engineering attention than it's getting: Artificial Analysis lists Mercury 2.5 from Inception at roughly 1,178 output tokens per second, ranked #2…
dev.to — LLM tag
TIER_1English(EN)·Mohd Yusuf Khan·
<p><em>Originally published on <a href="https://learn.homingo.co.in/guides/llm-agent-evals-traces-budgets-python" rel="noopener noreferrer">Homingo Learn</a>.</em></p> <p><em>An agent you cannot measure is an agent you cannot change safely. Three pieces make it measurable: traces…
