PulseAugur
EN
LIVE 18:19:53

LLM research probes faithfulness, sycophancy, and knowledge updates · 10 sources tracked

Recent research explores various facets of large language model (LLM) behavior and evaluation. One study introduces Normalized Simulatability Gain (NSG) to measure the faithfulness of LLM self-explanations, finding they significantly improve prediction of model behavior across various models like Gemini-3.*, GPT-5.2, and claude-4.5. Another paper investigates "sycophancy" in LLMs, revealing that task difficulty and model reasoning capabilities are more influential than pressure tactics in causing models to abandon correct answers. Further research delves into session-level contamination in LLMs, proposing protocols to assess how false premises are adopted, and introduces Conditional Accuracy Profiling (CAP) to diagnose LLM judge performance under different conditions. Additionally, new methods like EngramEdit aim to enable decoupled knowledge updates in LLMs, while others focus on improving process predictability through step-by-step execution and disentangling model effects from personas in multi-agent simulations. Finally, one study suggests that activation steering for debiasing LLMs may primarily reduce model confidence rather than address underlying fairness issues. AI

IMPACT These studies offer new tools and insights for evaluating LLM behavior, improving their reliability, and enabling more effective knowledge updates, crucial for advancing AI development and deployment.

RANK_REASON The cluster consists of multiple academic papers published on arXiv, detailing novel research methodologies and findings in LLM behavior and evaluation.

Read on Towards AI →

AI-generated summary · Google Gemini · from 174 sources. How we write summaries →

LLM research probes faithfulness, sycophancy, and knowledge updates · 10 sources tracked

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
The cluster consists of multiple academic papers published on arXiv, detailing novel research methodologies and findings in LLM behavior and evaluation.
Source corroboration
174 independent sources
Strong cross-source corroboration — multiple independent publishers covered this within the clustering window.
Topics
paper, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
53 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.
Coverage growth since scoring
+74 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

Full methodology in our editorial standards.

COVERAGE [174]

  1. arXiv cs.AI TIER_1 English(EN) · Junchi Liao ·

    Monitor-Validity Failures: LLM Safety Evidence Paths at Evolving Framework Boundaries

    arXiv:2610.23362v1 Announce Type: new Abstract: Modern LLM applications rely on runtime monitors that inspect tool calls, traces, streams, detector outputs, and verifier inputs. This creates an interface-reliability problem: a framework path may remain operational while policy-re…

  2. arXiv cs.CL TIER_1 English(EN) · Yinuo Xu, Jadie Adams, Brian Hu, Shivani Kumar, David Jurgens, Arslan Basharat ·

    VISTA: Steering LLMs with Latent User Values Inferred from a Few Survey Answers

    arXiv:2610.23702v1 Announce Type: new Abstract: Personalizing a language model requires knowing what a person values, which stays largely consistent across tasks. Yet most methods either learn task-specific preferences or place the user's profile in the prompt, where models strug…

  3. arXiv cs.CL TIER_1 English(EN) · Zesheng Shi, Xue Li, Chenrui Dai, Jing Li ·

    Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data

    arXiv:2610.22874v1 Announce Type: new Abstract: LLMs over external data face a per-instruction tension: a single document or tool return often interleaves legitimate directives with adversarial prompt injections, and the model must follow the former while rejecting the latter. Ex…

  4. arXiv cs.AI TIER_1 English(EN) · Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Chenhao Huang, Tao Gui, Qi Zhang, Xuanjing Huang ·

    Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

    arXiv:2505.10597v3 Announce Type: replace-cross Abstract: Reward models (RMs) are central to aligning large language models (LLMs) with human values, yet noisy preference feedback can induce reward misgeneralization through spurious correlations and overfitting. We characterize t…

  5. arXiv cs.AI TIER_1 English(EN) · Licheng Pan, Haocheng Yang, Haoxuan Li, Yuzhou Chen, Yichen Sun, Yuan Lu, Xiaoxi Li, Hao Wang, Zhixuan Chu, Kui Ren ·

    Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

    arXiv:2610.21214v1 Announce Type: new Abstract: Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a binary judgment from an LLM verifier. However, judgment certainty varies substant…

  6. arXiv cs.AI TIER_1 English(EN) · Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe, Khuushi Maheshwari ·

    Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence

    arXiv:2610.22139v1 Announce Type: new Abstract: Confidence scores determine whether an LLM answer is accepted, reviewed, or regenerated, yet scalar recalibration can only correct errors visible through the score itself. We introduce RADAR (Repair, Adopt, and Diagnose After Recali…

  7. arXiv cs.AI TIER_1 English(EN) · Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang, Xiangru Tang, Jiaxuan You, Heng Ji ·

    JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

    arXiv:2610.21412v1 Announce Type: new Abstract: Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls …

  8. arXiv cs.CL TIER_1 English(EN) · Sara Rezaeimanesh, Chad Klochko, Kundan Thind, Mohammad Ghassemi ·

    INCONVENIENT: A Benchmark to Assess LLM Generalization Capabilities from Convenience Samples

    arXiv:2610.21038v1 Announce Type: cross Abstract: Machine learning models for tabular prediction typically assume that training data are representative of the population on which they will be deployed. In practice, however, data are often collected based on convenience, availabil…

  9. arXiv cs.AI TIER_1 English(EN) · Alexey A. Bokov, Ivan I. Panin, Nikolay B. Nechaev, Mikhail Yu. Valkov, Valeria Yu. Chernina, Natalia Gombolevskaya, Ayyaz Hussain, Vladimir M. Agafonov, Victor A. Gombolevskiy, Aleksandr V. Nesterov, Iaroslav R. Bespalov, Ivan V. Oseledets, Dmitry V. Dy… ·

    AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

    arXiv:2610.22298v1 Announce Type: cross Abstract: Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clinically coherent, actionable, and safe. We developed a multidimensional benchmark …

  10. arXiv cs.AI TIER_1 English(EN) · Owen Walters, Lihu Chen, Francesca Toni ·

    Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence

    arXiv:2610.21731v1 Announce Type: cross Abstract: Web search allows LLMs to access up-to-date information, but it remains unclear why they sometimes produce outdated answers even when relevant evidence is available. To study this problem, we construct temporal facts whose values …

  11. arXiv cs.AI TIER_1 English(EN) · Jianchang Su, Dawei Xiang, Yifan Zhang, Wei Zhang ·

    Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation

    arXiv:2610.21135v1 Announce Type: cross Abstract: LLM-agent evaluation reports behavioral scores but leaves the identity of the deployed runtime stack unreported. A tampered deployment that preserves evaluated outputs passes every output-only evaluator. We introduce DIB-Bench, a …

  12. arXiv cs.IR (Information Retrieval) TIER_1 English(EN) · Francesca Toni ·

    Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence

    Web search allows LLMs to access up-to-date information, but it remains unclear why they sometimes produce outdated answers even when relevant evidence is available. To study this problem, we construct temporal facts whose values changed after the models' training cutoffs. We eva…

  13. Hugging Face Daily Papers TIER_1 English(EN) ·

    JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

    Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which…

  14. arXiv cs.AI TIER_1 English(EN) · Jianlin Chen, Wenhui Chen, Chi Man Vong ·

    Visible but Not Avoided: State-Dependent Re-Proposal of Rejected Candidates in Catalog-Based LLM Optimization Loops

    arXiv:2610.15738v1 Announce Type: new Abstract: LLM optimization loops often show the proposer the candidates it has already tried and rejected, but leave it to the proposer to avoid them. We ask whether avoidance measured in a static probe predicts avoidance in the states an opt…

  15. arXiv cs.AI TIER_1 English(EN) · Ayoub Merimi, Souhail Bakkali, Abdeljalil Sakat ·

    Fresh at Decision, Stale at Action: Routing New Information to LLM Decisions Already Made

    arXiv:2610.18126v1 Announce Type: new Abstract: Freshness shapes an LLM decision at the moment it is made; the action it commits to is latent, carried out later, in a changing world. We therefore introduce a timestamp theory of decision validity and a decision ledger that realise…

  16. arXiv cs.AI TIER_1 English(EN) · Jingxiang Meng, Aofan Liu, Yaoshuai Ma, Fangxin Liu, Yongbiao Chen ·

    REA: From Correct LLM Judgments to Safe Decision Delivery

    arXiv:2610.17473v1 Announce Type: new Abstract: Generative safety guards must classify requests correctly and return outputs that applications can parse. Reasoning guards can improve classification accuracy while producing outputs that applications cannot parse. When applications…

  17. arXiv cs.AI TIER_1 English(EN) · Kounianhua Du, Jianxing Liu, Kangning Zhang, Wenxiang Jiao, Yuan Lu, Jiarui Jin, Jianghao Lin, Weiwen Liu, Yong Yu, Weinan Zhang ·

    FleetPilot: Fine-Grained Steering Empowers Real-World LLM Personalization

    arXiv:2610.14970v1 Announce Type: new Abstract: With the rapid evolution of large language models (LLMs), there is an increasing demand for effective personalization techniques that can adapt model behavior to individual user preferences. Personalized LLMs must infer a user's que…

  18. arXiv cs.LG TIER_1 English(EN) · Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S Dhillon, Rishabh Agarwal, Devvrit Khatri ·

    Learning, Fast and Slow: Towards LLMs That Adapt Continually

    arXiv:2605.12484v3 Announce Type: replace Abstract: Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating model parameters forces them to absorb task-specific information, which can result in catastrophic forg…

  19. arXiv cs.CL TIER_1 English(EN) · Shuo Xing, Junyuan Hong, Yifan Wang, Runjin Chen, Zhenyu Zhang, Ananth Grama, Zhengzhong Tu, Zhangyang Wang ·

    LLMs Can Get "Brain Rot": A Pilot Study on Twitter/X

    arXiv:2510.13928v3 Announce Type: replace Abstract: We propose and test the LLM Brain Rot Hypothesis: continual exposure to junk web text induces lasting cognitive decline in large language models (LLMs). To unveil junk effects, we designed a novel controlled experiment on real T…

  20. arXiv cs.CL TIER_1 English(EN) · Gunhee Cho, Yun-Gyung Cheong ·

    Scaling Personality Control in LLMs with Big Five Scaler Prompts

    arXiv:2508.06149v2 Announce Type: replace Abstract: We present Big5-Scaler, a prompt-based framework for conditioning large language models (LLMs) with controllable Big Five personality traits. By embedding numeric trait values into natural language prompts, our method enables fi…

  21. arXiv cs.CL TIER_1 English(EN) · Minsuh Joo, Youna Kim, Choonghyun Park, Soo Kyung Kim, Hyunsoo Cho ·

    MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation

    arXiv:2610.19827v1 Announce Type: cross Abstract: LLMs absorb vast knowledge from training data, yet their inherently probabilistic nature offers no guarantee that this knowledge will surface in their outputs. To bridge this gap, Knowledge Extraction Enhancement (KEE) techniques …

  22. arXiv cs.CL TIER_1 English(EN) · Minchan Kim, Sungho Bae, Youngjin Kim, Hyunwoo Park ·

    Where to Start, How to Move: Analyzing Curricula under GRPO for Long-Horizon LLM Agents

    arXiv:2610.19349v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) provides no reward-relative learning signal when all rollouts receive identical rewards. In long-horizon tasks that require multiple interactions, rewards are given only upon successful ta…

  23. arXiv cs.CL TIER_1 English(EN) · Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio Andr\'e Machado Porto ·

    Manac\'a-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge

    arXiv:2610.20570v1 Announce Type: new Abstract: Brazilian Portuguese remains underserved by open-source language models, and the few that are safety-aligned rarely document what did not work or validate comparisons with statistical significance. We present Manac\'a-1B-Instruct-2,…

  24. arXiv cs.CL TIER_1 English(EN) · Yibin Lou, Yi Hu, Jianhui Chen, Chenpeng Wang, Muhan Zhang, Liangming Pan ·

    Which Knowledge Should LLMs Trust? Uncovering a Generalizable Steering Direction Across Knowledge Conflicts

    arXiv:2610.20067v1 Announce Type: new Abstract: Large Language Models (LLMs) frequently encounter conflicts between internal parametric knowledge and external contextual knowledge, yet the mechanisms governing this selection remain unclear. We show that the choice between context…

  25. arXiv cs.CL TIER_1 English(EN) · Haocheng Yang, Licheng Pan, Yiming Dong, Xiaoxi Li, Yuan Lu, Zhenhao Chen, Jiajun Fan, Shuai Shao, Chunyuan Zheng, Hao Wang ·

    Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

    arXiv:2610.19915v1 Announce Type: new Abstract: Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verif…

  26. arXiv cs.AI TIER_1 English(EN) · Chun-Hao Yang, Bo-Han Feng, Tzu-Yuan Lai, Yan-Yu Chen, Yin-Kai Dean Huang, Shou-De Lin ·

    Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap

    arXiv:2511.00198v2 Announce Type: replace-cross Abstract: Autoregressive language models are usually trained with next-token prediction under a fixed left-to-right target order. This paper studies target-token organization as a training-time design choice across arithmetic reason…

  27. arXiv cs.AI TIER_1 English(EN) · Durga Prasad Maram, Dhruvin Gandhi, Zonghai Yao, Gayathri Akkinapalli, Franck Dernoncourt, Yu Wang, Ryan A. Rossi, Nesreen K. Ahmed ·

    Iterative Critique-Refine Framework for Enhancing LLM Personalization

    arXiv:2510.24469v2 Announce Type: replace-cross Abstract: Personalized text generation requires models not only to produce coherent text but also to align with a target user's style, tone, and topical focus. Existing retrieval-augmented approaches such as LaMP and PGraphRAG enric…

  28. arXiv cs.AI TIER_1 English(EN) · Jasper Dekoninck, Maximilian Baader, Martin Vechev ·

    Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation

    arXiv:2409.00696v4 Announce Type: replace-cross Abstract: Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs). However, current rating systems suffer from several important limitations: first…

  29. arXiv cs.AI TIER_1 English(EN) · Bharath Kumar Bolla, Chaitali Deb Purkayastha, Vishnu Surya Reddy Nandi ·

    Right Answer, Wrong Reason: Accuracy, Consistency, and Consensus Are Misleading Indicators of LLM Faithfulness in Clinical Decision Support

    arXiv:2609.32817v2 Announce Type: replace Abstract: Clinical Large Language Models (LLMs) achieve strong medical-exam accuracy; however, a correct answer does not guarantee that the explanation names the concepts that actually drove the decision. We introduce three lightweight, d…

  30. arXiv cs.AI TIER_1 English(EN) · Yifan He, Manon Reusens, Sofie Goethals, David Martens ·

    A Global View of How LLMs Value Human Time

    arXiv:2610.18515v2 Announce Type: cross Abstract: Large language models (LLMs) increasingly act as personal assistants; however, little is known about how they capture national-level preferences on a global scale. Given that "time" is a universal and representative concept, we ex…

  31. arXiv cs.AI TIER_1 English(EN) · Pratuat Amatya, Vinay Setty, V Venktesh, Avishek Anand ·

    Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison

    arXiv:2610.18415v1 Announce Type: cross Abstract: Large Language Models (LLMs) are aligned to human preferences, after pre-training and supervised fine-tuning steps. However, standard RL alignment steps such as RLHF do not focus on factuality of LLMs, hence they may generate non-…

  32. arXiv cs.AI TIER_1 English(EN) · Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli, Lorenzo Perini, Niek Tax, Daniel Haimovich, Jesse Davis, Andrea Passerini, Andrea Pugnana ·

    A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

    arXiv:2610.18108v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgme…

  33. arXiv cs.AI TIER_1 English(EN) · Gonzalo Mancera, Julian Fierrez, Aythami Morales, Ruben Tolosana, Daniel DeAlcala, Francisco Jurado ·

    inLoRA-MINT: Internal Representations for Auditing Training Data in Domain-adapted LLMs

    arXiv:2610.18051v1 Announce Type: cross Abstract: This work investigates the application of the Membership Inference Test (MINT) to large language models (LLMs) fine-tuned with LoRA, aiming to determine whether specific samples were used during training and address concerns about…

  34. arXiv cs.AI TIER_1 English(EN) · Vasileios Saketos, William Reveillard, Alexandre Proutiere, Sebastian Dalleiger, Ming Xiao ·

    To Defer or Not to Defer? Online Learning with LLMs and Congested Experts

    arXiv:2610.15128v1 Announce Type: cross Abstract: Modern AI systems increasingly combine large language models (LLMs) with human experts, using the LLM to handle routine queries while deferring uncertain cases for expert review. When expert capacity is limited, however, deferral …

  35. arXiv cs.AI TIER_1 English(EN) · Yang Xiao, Yusong Sun, Haoyi Wu, Wenyang Hui, Weiqiang Hang, Luqing Wang, Zejian Wang, Zirui Tu, Jingsheng Mao, Shaobo Hu, Yang Luo, Jiale Guo, Jin Zhang, Yudan Liu, Zhaokai Luo, Ren Huan, Yao Hu, Chengyue Jiang, Wenjie Li, Chuan Mu ·

    Vela: User Behavior as Supervision Sparks LLM Intelligence

    arXiv:2610.14709v1 Announce Type: cross Abstract: Large language models are increasingly deployed in applications that act on users' long-term behavior histories, from personal assistants to recommendation and advertising systems. These applications demand the user dimension of L…

  36. Hugging Face Daily Papers TIER_1 English(EN) ·

    Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation

    LLM-agent evaluation reports behavioral scores but leaves the identity of the deployed runtime stack unreported. A tampered deployment that preserves evaluated outputs passes every output-only evaluator. We introduce DIB-Bench, a benchmark that measures this deployment-integrity …

  37. Hugging Face Daily Papers TIER_1 English(EN) ·

    JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

    A harness that delegates agent tool choices to small decision models via fitted confidence thresholds reduces cost and latency while preserving task success, supported by a new benchmark and a 4B model trained on agent error states.

  38. Hugging Face Daily Papers TIER_1 English(EN) ·

    Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports

    LLM-generated crisis reports match human performance on some metrics but fail to supply professional provenance and reproduce few official figures, while common automatic scores poorly reflect completeness.

  39. Hugging Face Daily Papers TIER_1 English(EN) ·

    Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison

    Large Language Models (LLMs) are aligned to human preferences, after pre-training and supervised fine-tuning steps. However, standard RL alignment steps such as RLHF do not focus on factuality of LLMs, hence they may generate non-factual information, which could result in uninten…

  40. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Abdeljalil Sakat ·

    Fresh at Decision, Stale at Action: Routing New Information to LLM Decisions Already Made

    Freshness shapes an LLM decision at the moment it is made; the action it commits to is latent, carried out later, in a changing world. We therefore introduce a timestamp theory of decision validity and a decision ledger that realises it. Six timestamps define a decision; a freshn…

  41. arXiv cs.CL TIER_1 English(EN) · Feifei Qian, Xuan Chen, Lixin Cui, Chaoyang Mei, Fanlin Meng, Shuyao Xiao, Ming Cui, Junxi Yin, Lu Bai ·

    From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

    arXiv:2610.13375v1 Announce Type: new Abstract: Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In …

  42. arXiv cs.AI TIER_1 English(EN) · Jiwon Jeon, Myungsik Cho, Jeonghye Kim, Youngchul Sung ·

    Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction

    arXiv:2610.13672v1 Announce Type: cross Abstract: Large language models struggle to reliably identify and correct errors in their own reasoning. Unlike prior training-based approaches that fine-tune an LLM to generate revisions or feedback, we learn where to intervene in failed r…

  43. Hugging Face Daily Papers TIER_1 English(EN) ·

    Vela: User Behavior as Supervision Sparks LLM Intelligence

    Large language models are increasingly deployed in applications that act on users' long-term behavior histories, from personal assistants to recommendation and advertising systems. These applications demand the user dimension of LLM intelligence: inferring a person's preferences,…

  44. arXiv cs.AI TIER_1 English(EN) · Zhaoxin Yu, Qingchao Kong, Dajun Zeng, Wenji Mao ·

    Examining Social Attribution in LLM Reasoning: A Theory-Guided Probing Methodology

    arXiv:2610.12022v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed in sociotechnical systems where social attribution, the reasoning process attributing external events to the causes and reasons of agents' social behaviors, plays a critical r…

  45. arXiv cs.AI TIER_1 English(EN) · Andrea Wynn (Nathan), Harsh Satija (Nathan), Seokhyun (Nathan), Baek, Anqi Liu, Eric Nalisnick, Gillian K. Hadfield ·

    Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs

    arXiv:2610.10906v1 Announce Type: new Abstract: Human communities are governed by normative systems: shared standards that produce \textit{norms} dictating acceptable behavior, enforced through community sanctioning. Aligning increasingly autonomous AI systems with these norms is…

  46. arXiv cs.AI TIER_1 English(EN) · Zheng Huang, Sansheng Cao, Enpei Zhang, Weikang Qiu, Elynn Chen, Xiang Zhang, Yaoqing Yang, Rex Ying, Dawei Zhou, Yujun Yan ·

    Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance

    arXiv:2610.12304v1 Announce Type: new Abstract: Understanding large language model (LLM) reasoning requires looking beyond behavioral performance to examine how reasoning ability is reflected in internal organization. Inspired by neuroscience findings linking higher intelligence …

  47. arXiv cs.AI TIER_1 English(EN) · Haitong Jiang, Chunlin Liu, Sihan Tang, Chan Wu, Xiaoqing Su, Yuhong Feng ·

    Same Outcome, Different Evidence: Intent Recovery in LLM Safety Evaluation

    arXiv:2610.11766v1 Announce Type: cross Abstract: Safety evaluations of large language models commonly summarize harmful-output behavior with attack success rate (ASR). Yet the same non-harmful outcome can arise for very different reasons. A model may recover a harmful task and r…

  48. arXiv cs.CL TIER_1 English(EN) · Vihindi Kotalawala, Nevidu Jayatilleke ·

    PlurVA-LLM-2026 Shared Task Track-1: Pluralistic Value Alignment in LLMs via Multilingual Fine-Tuning and Threshold Calibration

    arXiv:2609.32382v2 Announce Type: replace Abstract: We present our system for the PlurVA-LLM 2026 Shared Task Track-1, which focuses on pluralistic value alignment in the contexts of China, Indonesia, and Sri Lanka. For this resource-constrained track, we fine-tuned Llama 3.1 8B …

  49. arXiv cs.CL TIER_1 English(EN) · Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu, Shenlong Zhang, Yijia Zhang, Ling Luo, Hongfei Lin ·

    Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

    arXiv:2610.11978v1 Announce Type: new Abstract: Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13 multi…

  50. arXiv cs.AI TIER_1 English(EN) · Yuren Hao, Xiang Wan, ChengXiang Zhai ·

    An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems

    arXiv:2508.08833v4 Announce Type: replace-cross Abstract: Frontier large language models (LLMs) now reach near-ceiling accuracy on standard mathematical-reasoning benchmarks and gold-medal-level performance at the International Mathematical Olympiad. As these benchmarks saturate …

  51. arXiv cs.IR (Information Retrieval) TIER_1 English(EN) · Qingyao Ai ·

    Overview of the NTCIR-19 Automatic Evaluation of LLMs 2 (AEOLLM-2) Task

    In this paper, we provide an overview of the NTCIR-19 Automatic Evaluation of LLMs 2 (AEOLLM-2) task. Building on the success of the NTCIR-18 core task AEOLLM, we proposed AEOLLM-2 for NTCIR-19 to further investigate automatic evaluation methods for Large Language Models (LLMs), …

  52. arXiv cs.CL TIER_1 English(EN) · Fahrell Giovanny, Geby Bayuningtyas, Sahrul Mukharom, Hafiz Budi Firmansyah ·

    Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigation

    arXiv:2609.35308v2 Announce Type: replace Abstract: Large language models treat conversation history as unverified context, so false premises injected into prior turns can be adopted as fact, a failure mode we term session-level contamination. We introduce five contamination prot…

  53. arXiv cs.LG TIER_1 English(EN) · Wenqi Li, Bin Liu, Mindi Ruan, Chuanbo Hu, Minglei Yin, Xin Li ·

    Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions

    arXiv:2610.09229v1 Announce Type: new Abstract: LLM-as-judge is now a standard tool for scalable evaluation, but judge performance is still often summarized by a single accuracy number. This aggregate view hides the deployment conditions under which a judge succeeds or fails. We …

  54. arXiv cs.CL TIER_1 English(EN) · Xiaoshu Chen, Sihang Zhou, Ke Liang, Xinwang Liu ·

    SeOPD: Self-Evolving LLMs via Online Policy Distillation from Self-Generated Chain-of-Thought

    arXiv:2609.33181v2 Announce Type: replace-cross Abstract: Recent advances in online policy self-distillation (OPSD) have demonstrated that large language models (LLMs) can improve their capabilities by leveraging external privileged information (PI), such as manual annotations or…

  55. arXiv cs.CL TIER_1 English(EN) · Hongru Cai, Ran Wei, Wenjie Wang, Chengfa Wu, Ning Song, Yongqi Li, Wenjie Li ·

    EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

    arXiv:2610.10533v1 Announce Type: new Abstract: Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this archite…

  56. arXiv cs.CL TIER_1 English(EN) · Guang Yang, Homa Hosseinmardi, Fengchen Liu, Amir Ghasemian ·

    Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding

    arXiv:2610.08840v1 Announce Type: new Abstract: Large language models (LLMs) often abandon a correct answer, or endorse a user's position, once the user pushes back. This behavior, called sycophancy, is usually reported as a single rate per model, which says little about when it …

  57. arXiv cs.LG TIER_1 English(EN) · Harry Mayne, Justin Singh Kang, Dewi Gould, Kannan Ramchandran, Adam Mahdi, Noah Y. Siegel ·

    A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior

    arXiv:2602.02639v2 Announce Type: replace-cross Abstract: LLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the model's true reasoning process is poorly understood. Existing faithfulness metrics have critical limitations, ty…

  58. Hugging Face Daily Papers TIER_1 English(EN) ·

    Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs

    Human communities are governed by normative systems: shared standards that produce \textit{norms} dictating acceptable behavior, enforced through community sanctioning. Aligning increasingly autonomous AI systems with these norms is a central alignment challenge, complicated by t…

  59. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Gillian K. Hadfield ·

    Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs

    Human communities are governed by normative systems: shared standards that produce \textit{norms} dictating acceptable behavior, enforced through community sanctioning. Aligning increasingly autonomous AI systems with these norms is a central alignment challenge, complicated by t…

  60. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Gillian K. Hadfield ·

    Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs

    Human communities are governed by normative systems: shared standards that produce \textit{norms} dictating acceptable behavior, enforced through community sanctioning. Aligning increasingly autonomous AI systems with these norms is a central alignment challenge, complicated by t…

  61. arXiv cs.CL TIER_1 English(EN) · Sofia Torres, Gabriel Almeida, Carter Adams, Camila Rocha ·

    When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPO

    arXiv:2610.06861v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has become the dominant paradigm for eliciting multi-step reasoning in large language models, and a recent wave of methods (LUFFY, ExPO, PAPO, TAPO) further augments RL with \e…

  62. arXiv cs.CL TIER_1 English(EN) · Zizhuo Zhang, Xiong Peng, Jingwei Sun, Rong Yao, Borui Jiang, Bo Han ·

    Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspective

    arXiv:2610.07894v1 Announce Type: new Abstract: Large language models (LLMs) are expected to answer questions faithfully based on the provided context, abstaining when the context information is insufficient to answer the questions. Existing faithfulness evaluations typically ass…

  63. arXiv cs.CL TIER_1 English(EN) · Kyojun Choo, Minsoo Song, Yunju Kang, Chanjun Park ·

    Structured but Silent: Probing Capability Requirements in LLM Hidden States

    arXiv:2610.08018v1 Announce Type: new Abstract: Reliable tool use requires more than triggering a mechanism or matching a query to an API description. Before selecting a specific tool, an agent must first infer the capability requirements implied by the user query. In this paper,…

  64. arXiv cs.CL TIER_1 English(EN) · Pierre-Carl Langlais, Pavel Chizhov, Catherine Arnett, Carlos Rosas-Hinostroza, Mattia Nee, Eliot Krzystof Jones, Ir\`ene Girard, David Mach, Anastasia Stasenko, Ivan P. Yamshchikov ·

    Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

    arXiv:2506.01732v4 Announce Type: replace Abstract: Large Language Models (LLMs) are pre-trained on large amounts of data from different sources and domains. Such datasets often contain trillions of tokens, including large portions of copyrighted or proprietary content, which rai…

  65. arXiv cs.LG TIER_1 English(EN) · Yuhwan Jeong, Jinnyeong Yang, Kuk-Jin Yoon ·

    Do LLMs Act on What They Know? From Partner Representations to Cooperative Actions

    arXiv:2610.08129v1 Announce Type: new Abstract: Cooperation with unfamiliar partners requires adapting to communication conventions that are not known in advance. We study this problem in a controlled Hanabi-derived environment with scripted hint generation, LLM-controlled receiv…

  66. arXiv cs.LG TIER_1 English(EN) · Chen Chen, Dongjie Wang, Mei Liu, Zijun Yao ·

    Cite What You Explore: Budget-Aware LLM Reasoning over Medical KGs with Verifiable Evidence

    arXiv:2610.07739v1 Announce Type: new Abstract: Post-discharge risk prediction from electronic health records (EHRs) is difficult because many dependencies that link discharge-time observations to downstream complications, such as comorbidity cascades and drug-disease interaction…

  67. arXiv cs.AI TIER_1 English(EN) · Bach Nguyen, Zhaonan Li, Mau Son Nguyen, Sanika Chavan, Nilay Kumar, Hong Anh Nguyen, Khoa Vo, Ben Zhou ·

    How Well Do LLMs Reason with Noisy Evidence? An Active Visual Reasoning Benchmark

    arXiv:2610.07751v1 Announce Type: new Abstract: Real-world reasoning rarely reduces to static question answering: agents must actively gather information from tools and sensors that are often noisy and unreliable. Yet most existing active reasoning benchmarks assume that environm…

  68. arXiv cs.AI TIER_1 English(EN) · Maoqi Liu, Quan Fang, Yufei He ·

    CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

    arXiv:2610.08312v1 Announce Type: new Abstract: Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-L…

  69. arXiv cs.AI TIER_1 English(EN) · Bingnan Xiao, Chenhao Yang, Bingcong Li, Wei Ni, Xin Wang ·

    Adaptive Power Sampling for LLM Reasoning

    arXiv:2610.08563v1 Announce Type: new Abstract: Sequence-level power sampling has recently emerged as a training-free approach to reasoning by sampling from a sharpened output distribution of a base large language model (LLM). Nevertheless, existing methods typically sharpen the …

  70. arXiv cs.AI TIER_1 English(EN) · Weixian Xu, Yanzhe Zhang, Zora Zhiruo Wang, Changyu Chen, Diyi Yang ·

    Sherpa: Teaching LLMs to Teach Adaptively

    arXiv:2610.08778v1 Announce Type: new Abstract: Large language models (LLMs) have become increasingly capable problem solvers, but being able to solve a problem is not the same as being able to teach it. Existing approaches to training LLMs as teachers rely on demonstrations, pre…

  71. arXiv cs.AI TIER_1 English(EN) · Hashmath Shaik, Gnaneswar Villuri, Alex Doboli ·

    Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations

    arXiv:2610.07115v1 Announce Type: cross Abstract: The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whe…

  72. arXiv cs.AI TIER_1 English(EN) · Wonjun Lee, Kyungsik Yang, Gaeun Ji, Vaidehi Patil, Haon Park, Bumsub Ham, Mohit Bansal, Suhyun Kim ·

    Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge

    arXiv:2610.07532v1 Announce Type: cross Abstract: LLMs have advanced rapidly, raising growing concerns about their safety. Recent work has proposed approaches to detect and defend against attacks including defenses at decoding stage that leverage models' hidden states. However, e…

  73. arXiv cs.AI TIER_1 English(EN) · Dani Roytburg, Daphne Ippolito ·

    Disentangling Models from Personas in Heterogeneous LLM Simulations

    arXiv:2610.07535v1 Announce Type: cross Abstract: Multi-agent simulations with large language models (LLMs) often operate networks of agents with a single base model. This overlooks the inter-model effects which may dominate engagement dynamics in real-world deployments. To show …

  74. arXiv cs.AI TIER_1 English(EN) · Hans Schabert, Christoph Peters ·

    One Step at a Time: Trading LLM Autonomy for Process Predictability

    arXiv:2610.07817v1 Announce Type: cross Abstract: Organizations automating operational processes need more than a correct outcome: they need to predict how a process will run, know which one actually ran, and inspect it step by step. When an agent is the executor that predictabil…

  75. arXiv cs.AI TIER_1 Français(FR) · Stephanie Buttigieg, Maeve Madigan, Parameswaran Kamalaruban, Stuart Burrell ·

    Latent space bias directions in LLMs capture confidence, not fairness

    arXiv:2610.08559v1 Announce Type: cross Abstract: Activation steering has gained popularity as a lightweight inference-time debiasing technique for large language models. However, prior work reports that steering vectors generalise poorly, with unintended effects on model perform…

  76. arXiv cs.CL TIER_1 English(EN) · Yiqi Liu, Joseph James, Yang Wang, Kun Zhao, Chenghao Xiao, Chenghua Lin ·

    JudgeMoE: Distributional Aggregation for LLM-as-a-Judge

    arXiv:2610.07109v1 Announce Type: new Abstract: When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights…

  77. Hugging Face Daily Papers TIER_1 English(EN) ·

    EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

    Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple…

  78. arXiv cs.IR (Information Retrieval) TIER_1 English(EN) · Paul Mutawe ·

    From High Recall to High Utility: Dataset-Adaptive Post-Processing of LLM-Generated Customer Intents

    Large language models can extract useful signals from heterogeneous enterprise data, but high-recall extraction often produces outputs that are duplicated, uneven in granularity, semantically overlapping, or too numerous for downstream systems and human reviewers to use effective…

  79. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Daphne Ippolito ·

    Disentangling Models from Personas in Heterogeneous LLM Simulations

    Multi-agent simulations with large language models (LLMs) often operate networks of agents with a single base model. This overlooks the inter-model effects which may dominate engagement dynamics in real-world deployments. To show this, we simulate a heterogeneous social network p…

  80. Hugging Face Daily Papers TIER_1 English(EN) ·

    Sherpa: Teaching LLMs to Teach Adaptively

    Large language models (LLMs) have become increasingly capable problem solvers, but being able to solve a problem is not the same as being able to teach it. Existing approaches to training LLMs as teachers rely on demonstrations, preference data, or predefined pedagogical criteria…

  81. arXiv cs.LG TIER_1 English(EN) · Dario Vajda ·

    Teaching LLMs to See Graphs: Unifying Text and Structural Reasoning

    arXiv:2605.10247v2 Announce Type: replace Abstract: Applying Large Language Models (LLMs) to graph-structured data usually involves multi-step pipelines in which textual node attributes are compressed into single tokens and further processed by GNNs, discarding most of their sema…

  82. arXiv cs.CL TIER_1 English(EN) · Siyu Wang, Yifan Wang, Yuecheng He ·

    MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

    arXiv:2610.03080v1 Announce Type: cross Abstract: Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent: generated code runs, a backtest plots, yet the risk logic that the trader descr…

  83. arXiv cs.AI TIER_1 English(EN) · Louie Hong Yao, Vishesh Anand, Yuan Zhuang, Tianyu Jiang ·

    Rhetorical Questions in LLM Representations: A Linear Probing Study

    arXiv:2604.14128v3 Announce Type: replace-cross Abstract: Rhetorical questions are asked not to seek information but to persuade or signal stance. How large language models internally represent them remains unclear. We analyze rhetorical questions in LLM representations using lin…

  84. arXiv cs.AI TIER_1 English(EN) · Hongji Pu, Yilun Zhao, Wenpeng Yin ·

    PAPER2LLM++: Continual Self-Evolution of LLMs from Research Papers

    arXiv:2610.02793v1 Announce Type: new Abstract: Research on LLMs continually uncovers model limitations, their causes, and potential solutions. Yet these human discoveries remain largely disconnected from model evolution: an LLM does not automatically learn from new research abou…

  85. arXiv cs.AI TIER_1 English(EN) · Seoyeon Ye, Gayoung Kim, Jiyoung Hong, Sookyung Kim, Hyunsoo Cho ·

    LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs

    arXiv:2610.02902v1 Announce Type: new Abstract: Current analyses of LLMs' parametric knowledge are largely output-centric, drawing conclusions about what a model knows without verifying what it was actually trained on. This leaves fundamental questions, such as whether a correct …

  86. arXiv cs.AI TIER_1 English(EN) · Ritika Rekhi, Bing Zhang, Md Arman Islam, Jaya Krishna Pasham, Yeswanth Chitturi, Akshay Paramesha, Isha Valiveti, Asif Imran, Bekir Turkkan, Tevfik Kosar ·

    Improving the Energy-Efficiency of the Code Generated by LLMs through Effective Prompting

    arXiv:2610.02571v1 Announce Type: cross Abstract: As AI-assisted programming becomes increasingly mainstream, the environmental impact of AI-generated software has emerged as an important consideration. This motivates evaluating LLM-generated code beyond functional correctness by…

  87. arXiv cs.AI TIER_1 English(EN) · Katherine Van Koevering, Jon Kleinberg ·

    Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgments

    arXiv:2406.00092v2 Announce Type: replace Abstract: Randomness is central to human cognition and to many applications in which large language models are deployed, yet probabilistic token generation does not imply that LLMs can produce unbiased random sequences. We study how conte…

  88. arXiv cs.AI TIER_1 English(EN) · Iordanis Fostiropoulos, Muhammad Rafay Azhar, Abdalaziz Sawwan, Boyu Fang, Yuchen Liu, Jiayi Liu, Hanchao Yu, Qi Guo, Jianyu Wang, Fei Liu, Xiangjun Fan ·

    GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification

    arXiv:2603.29112v2 Announce Type: replace Abstract: We introduce GISTBench, a benchmark for evaluating Large Language Models' (LLMs) ability to understand users from their interaction histories in recommendation systems. Unlike traditional RecSys benchmarks that focus on item pre…

  89. arXiv cs.LG TIER_1 English(EN) · Seyedarmin Azizi, Erfan Baghaei Potraghloo, Minoo Ahmadi, Souvik Kundu, Massoud Pedram ·

    Power-SMC: Low-Latency Sequence-Level Power Sampling for Training-Free LLM Reasoning

    arXiv:2602.10273v3 Announce Type: replace-cross Abstract: Reasoning ability in large language models is often attributed to \emph{distribution sharpening}: concentrating output probability on high-likelihood sequences. Recent works show that this sharpening effect can be obtained…

  90. arXiv cs.CL TIER_1 English(EN) · Taiqiang Wu, Yuxin Cheng, Chenchen Ding, Runming Yang, Xincheng Feng, Wenyong Zhou, Zhengwu Liu, Ngai Wong ·

    Can We Trust LLMs on Memristors? Diving into Reasoning Ability under Non-Ideality

    arXiv:2603.13725v2 Announce Type: replace Abstract: Memristor-based analog compute-in-memory (CIM) architectures provide a promising substrate for the efficient deployment of Large Language Models (LLMs), owing to superior energy efficiency and computational density. However, the…

  91. Hugging Face Daily Papers TIER_1 English(EN) ·

    LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

    Jev-style decision models return categorical probability distributions over predefined options without generating free-form text, enabling software systems to act on their outputs directly. In this work, we investigate the extent to which general-purpose LLMs already possess this…

  92. Hugging Face Daily Papers TIER_1 English(EN) ·

    Do LLMs Understand Sequential Structure? A Controlled Study of Inference and Generation

    Large language models (LLMs) are increasingly used as interactive agents and simulators, yet it remains unclear whether they can recover latent sequential structure beyond surface action frequencies. This distinction is critical for behavioral simulation, where actions are often …

  93. arXiv cs.LG TIER_1 English(EN) · Sachin Gupta ·

    VERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence

    arXiv:2610.00833v1 Announce Type: cross Abstract: Fluent LLM explanations may not follow the evidence from a structured system. We present VERITYGATE, a four-gate checker for declared evidence IDs, entities, numbers, and claim types. It checks a fixed schema; it does not verify e…

  94. arXiv cs.AI TIER_1 English(EN) · Hongbo Chen, Guohua Lu, Ting Dang, Hong Jia ·

    Not Too Hard, Not Too Easy: Learning from Intermediate States for LLM Structured Reasoning

    arXiv:2609.33149v2 Announce Type: replace Abstract: A common principle of effective learning is to practice material that is neither already mastered nor too difficult to permit progress. We ask how to apply this principle to structured reasoning tasks such as Sudoku and maze sol…

  95. arXiv cs.AI TIER_1 English(EN) · Khawaja Murad ul Hassan, Mehran Ebrahimi ·

    Characterizing a Configuration Where Inference-Time PRM-Pruned Fragment Grafting Is Inert: Evidence from Three Reasoning LMs

    arXiv:2610.00047v1 Announce Type: new Abstract: Diversity collapse in parallel chain-of-thought has motivated inference-time interventions built on a natural design: when a process reward model (PRM) prunes a chain, its high-PRM prefix is extracted and grafted verbatim as an in-c…

  96. arXiv cs.AI TIER_1 English(EN) · Sajad Goudarzi, Samaneh Zamanifard, Seyed Amin Seyed Haeri, Moloud Nasiri, Hamed Rahimian ·

    Mathematical Transfer in LLMs Follows Reasoning Approach More Than Topic

    arXiv:2610.00331v1 Announce Type: new Abstract: When selecting mathematical training data for LLMs, a natural organizing principle is topic: probability examples for probability targets. An alternative is reasoning approach: worked solutions that share a solution method with the …

  97. arXiv cs.AI TIER_1 English(EN) · Nishtha N. Vaidya, Stephan Grimm, Thomas Hubauer, Thomas A. Runkler ·

    Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI

    arXiv:2610.00682v1 Announce Type: new Abstract: Large language models (LLMs) increasingly underpin scientific AI applications that reason over structured knowledge, from biomedical question answering to materials informatics. However, their logical reasoning often falls short, pr…

  98. arXiv cs.AI TIER_1 English(EN) · Yunfan Zhou, Ye Zhu, Zhihai Wang, Jianguo Yao, Haibing Guan, Xijun Li ·

    Certainty Is Not Just Correctness: Rethinking Token-Level Certainty in LLM Reasoning

    arXiv:2610.00296v1 Announce Type: cross Abstract: Token-level certainty is widely used as a proxy for correctness in LLM training and inference. However, the performance of certainty-based methods depends both on the information in certainty scores and on how those scores are use…

  99. arXiv cs.AI TIER_1 English(EN) · Matthieu Zimmer, Xiaotong Ji, Tu Nguyen, Haitham Bou-Ammar ·

    The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning

    arXiv:2610.00332v1 Announce Type: cross Abstract: Distilling the reasoning capabilities of large language models (LLMs) into smaller students is a central challenge for efficient deployment. Current approaches face a fundamental tension: optimizing purely for verifiable task rewa…

  100. arXiv cs.AI TIER_1 English(EN) · Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane, Sudip Mittal, Shahram Rahimi ·

    Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning

    arXiv:2610.00562v1 Announce Type: cross Abstract: Longitudinal clinical reasoning requires large language models (LLMs) to identify and integrate relevant evidence distributed across extended patient histories. Although long-context models can process increasingly large amounts o…

  101. arXiv cs.AI TIER_1 English(EN) · Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein, Rom Himelstein, Avi Mendelson, Amit LeVi ·

    Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs

    arXiv:2610.01428v1 Announce Type: cross Abstract: Generalization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed in different ways. Existing work typically evaluates generalization through aggre…

  102. arXiv cs.AI TIER_1 English(EN) · Tae-Eun Song ·

    When Does a Second Model Help? Cross-Model Review in LLM Verification

    arXiv:2610.01471v1 Announce Type: cross Abstract: Large language models now generate code, documentation, and analyses, and are increasingly used to review such output. We ask when a second review by a different model helps. Building on the author's earlier preprints, which varie…

  103. arXiv cs.AI TIER_1 English(EN) · Zichen Xie, Mrigank Pawagi, Lize Shao, Yang Hu, Wenxi Wang ·

    Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning

    arXiv:2610.01847v1 Announce Type: cross Abstract: Model specifications define how large language models (LLMs) should behave, guiding alignment training, inference-time behavior, and evaluation. Yet these specifications may themselves contain defects: two individually reasonable …

  104. arXiv cs.AI TIER_1 English(EN) · Ziyang Yu, Liang Zhao ·

    Distilling LLM Reasoning into Graph of Concept Predictors

    arXiv:2602.03006v3 Announce Type: replace Abstract: Deploying Large Language Models (LLMs) for discriminative workloads is often limited by inference latency, compute, and API costs at scale. Active distillation reduces these costs by querying an LLM oracle to train small discrim…

  105. arXiv cs.AI TIER_1 English(EN) · Vishvesh Bhat ·

    Program Learning with Verifiable Rewards: Symbolic Backpropagation for Post-Training LLMs

    arXiv:2608.28421v2 Announce Type: replace Abstract: Post training a language model to reason means updating its weights. Supervised finetuning and reinforcement learning both place the acquired capability inside the model where it cannot be inspected cannot be checked step by ste…

  106. arXiv cs.LG TIER_1 English(EN) · Chengzhu Bao, Xianglong Yan, Tianao Zhang, Jiaqi Chen, Shaoqiu Zhang, Yulun Zhang ·

    RATIO: Reasoning Analysis and Token-level Inference Optimization for Quantized Reasoning Models

    arXiv:2609.39801v1 Announce Type: new Abstract: Post-training quantization (PTQ) has become a widely adopted technique for reducing the memory footprint and inference cost of large language models (LLMs). However, recent studies reveal that when applied to reasoning models, PTQ n…

  107. arXiv cs.AI TIER_1 English(EN) · Guangsheng Yu, Litianyi Zhang, Qin Wang, Xu Wang, Mingyuan Li, Shaoxiong Ji, Ren Ping Liu, Massimo Piccardi ·

    Diversity Combining for Multi-Path LLM Reasoning

    arXiv:2609.38829v1 Announce Type: new Abstract: Multi-path reasoning methods such as self-consistency (SC) sample $K$ reasoning paths and choose the most frequent answer. However, their gains quickly plateau as $K$ increases, and existing methods do not predict when this saturati…

  108. arXiv cs.AI TIER_1 English(EN) · Olivia Macmillan-Scott, Mirco Musolesi ·

    ContextAdapt: Evaluating Contextual Adaptation and Value Alignment in LLMs

    arXiv:2609.38260v1 Announce Type: cross Abstract: Values such as honesty, autonomy, and confidentiality are often regarded as general principles underpinning AI alignment. However, what it means to act in accordance with these values can depend on the context in which a decision …

  109. arXiv cs.CL TIER_1 English(EN) · Bohan Zhang (Southeast University), Linan Yue (Southeast University), Weibo Gao (Hong Kong Polytechnic University), Pengyu Chen (Southeast University), Hong Guo (Southeast University), Yanqi Hao (ZTE Corporation) ·

    Offline Guidance, Online Reasoning: Reusing LLM Feedback for Small Language Models

    arXiv:2609.39346v1 Announce Type: new Abstract: Large language models (LLMs) offer strong reasoning capabilities but are often costly to access through commercial APIs, while small language models (SLMs) are easier to deploy locally yet remain weaker in reasoning. This capability…

  110. arXiv cs.AI TIER_1 English(EN) · Yuliana Shakhvalieva, Dmitrii Kharchev, Viacheslav Bezrukov, Inessa Fedorova, Dmitry Bocharov, Ivan Oseledets, Valerii Ternovskii ·

    What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling

    arXiv:2609.39967v1 Announce Type: cross Abstract: Recursive reasoning models apply a small shared Transformer block many times to refine a latent state. This gives them large effective depth with few parameters and makes them strong on algorithmic tasks. Such compact solvers are …

  111. arXiv cs.AI TIER_1 English(EN) · Yihuai Hong, Shauli Ravfogel, Chen Zhao, Eunsol Choi ·

    Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment

    arXiv:2609.38972v1 Announce Type: cross Abstract: Chain-of-thought (CoT) traces often serve as a proxy for how Large Language Models (LLMs) arrive at their answers. However, growing evidence shows that models' CoT often fails to reflect their internal computations and can be chan…

  112. arXiv cs.AI TIER_1 English(EN) · Qijia He, Yu Huang, Yuan Cheng, Yuxin Chen, Yingbin Liang ·

    Provable Test-Time Scaling for Beam Search in LLM Reasoning

    arXiv:2609.38672v1 Announce Type: cross Abstract: Beam-search-based test-time methods provide an effective way to improve large language model (LLM) performance on long-horizon generation by pruning invalid reasoning paths early, leading to significantly improved reasoning effici…

  113. arXiv cs.CL TIER_1 English(EN) · Michaela Regneri, Nina Scheller, S\"oren Laue ·

    Compact Language, Complex Model Shifts: How and Where Ambiguity and Underspecification Affect LLMs

    arXiv:2609.39572v1 Announce Type: new Abstract: We analyze how lexical ambiguity and underspecification affect language model training. We create artificial homonyms and artificial hypernyms as pseudowords and analyze the generative performance of language models as they are trai…

  114. Hugging Face Daily Papers TIER_1 English(EN) ·

    Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs

    Generalization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed in different ways. Existing work typically evaluates generalization through aggregate accuracy on a single prompt format, task, or …

  115. Hugging Face Daily Papers TIER_1 English(EN) ·

    What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling

    Recursive reasoning models apply a small shared Transformer block many times to refine a latent state. This gives them large effective depth with few parameters and makes them strong on algorithmic tasks. Such compact solvers are natural candidates for tools that an LLM can call …

  116. arXiv cs.AI TIER_1 English(EN) · Xianhui Zhang, Jian Yu, Chengyu Xie, Chenhang Cui, Shuyi Miao, Pengyang Shao, Yu Zheng, Fei Shen, Tat-Seng Chua ·

    actr: aligning thoughts and responses for multilingual safety in reasoning llms

    arXiv:2609.37054v1 Announce Type: new Abstract: Ensuring the safety of reasoning large language models (LLMs) across languages is essential for their reliable deployment. However, when exposed to jailbreak attacks in non-high-resource languages, these models may generate unsafe r…

  117. arXiv cs.AI TIER_1 English(EN) · Feiyang Li, Shengjing Liu, Qi Zhan, Sijie Cheng, Weiqing Wang, Hongwen Chen, Yuxuan Yang, Wen Wang, Yile Wang, Hui Huang ·

    Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs

    arXiv:2609.38070v1 Announce Type: new Abstract: As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming increasingly important for quantifying the uncertainty of their answers. Current meth…

  118. arXiv cs.AI TIER_1 English(EN) · Gurbir Arora, Toni J. B. Liu, Jiajun Bao, Rapha\"el Sarfati, Christopher J. Earls ·

    Causal and Interpretable Structures in LLM Compositional Tasks

    arXiv:2609.35970v1 Announce Type: cross Abstract: Large language models are able to solve tasks whose answers depend on not only individual input tokens, but also on relations among them. How is such relational information represented and processed across transformer layers? We s…

  119. arXiv cs.AI TIER_1 English(EN) · Ali Mohammadi Esfahani, Nafiseh Kahani, Samuel A. Ajila ·

    Complexity-Aware Evaluation of LLM Comprehension

    arXiv:2609.37405v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for software engineering tasks that require understanding existing source code, including behavior prediction, function explanation, debugging, and code review. However, aggregate…

  120. arXiv cs.CL TIER_1 English(EN) · Thomas Reiter, Christoph Kern, Fedor Miasnikov, Sofiia Nikolenko, Rob Chew, Stephanie Eckman, Frauke Kreuter ·

    Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation

    arXiv:2609.35824v1 Announce Type: new Abstract: Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets lab…

  121. arXiv cs.CL TIER_1 English(EN) · Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov, Carlos Rosas-Hinostroza, Neil Si Smail, Benjamin Burtin, Hanna Shcharbakova, Ivan Yamshchikov, Anastasia Stasenko ·

    It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

    arXiv:2609.37891v1 Announce Type: new Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--for instance, they contain little explicit reasoning. Thus, many frontier labs have…

  122. arXiv cs.CL TIER_1 English(EN) · Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen, Xi Ye ·

    LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

    arXiv:2609.38137v1 Announce Type: new Abstract: Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated acc…

  123. arXiv cs.CL TIER_1 English(EN) · Md. Ismail Hossain, Humaira Kousar, Isidora Chara Tourni ·

    Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning

    arXiv:2609.37915v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies this supervision to unverified student rollouts while conditioning the teacher …

  124. arXiv cs.CL TIER_1 English(EN) · Amir Taubenfeld, Zorik Gekhman, Lior Nezry, Omri Feldman, Natalie Harris, Shashir Reddy, Romina Stella, Ariel Goldstein, Marian Croak, Yossi Matias, Amir Feder ·

    Evaluating Alignment of Behavioral Dispositions in LLMs

    arXiv:2602.11328v2 Announce Type: replace Abstract: As people turn to LLMs for social advice, understanding their behavior in such contexts becomes essential. In this work, we focus on behavioral dispositions: the underlying tendencies that shape responses in social contexts. We …

  125. arXiv cs.LG TIER_1 English(EN) · Shenghong Dai, Shiva Kumar Pentyala, Yingchi Liu, Shubham Mehrotra, Suman Banerjee, James Zhu, Bin Bi, Sitaram Asur, Phil Mui ·

    Koa-action: Fast and Consistent Structured Decision Making with Generative LLMs

    arXiv:2609.36115v1 Announce Type: new Abstract: Industry applications often demand low-latency classification, yet current large language model (LLM) approaches remain poorly suited for latency-critical applications. Existing prompting and constrained decoding produce verbose, mu…

  126. arXiv cs.LG TIER_1 English(EN) · Qihao Wen, Jiahao Wang, Yang Nan, Pengfei He, Ravi Tandon, Han Xu ·

    Embedding Perturbation may Better Reflect Intermediate-Step Uncertainty in LLM Reasoning

    arXiv:2602.02427v3 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved significant breakthroughs across various domains, but they can still produce unreliable or misleading outputs. For responsible LLM applications, uncertainty quantification techniques ar…

  127. arXiv cs.AI TIER_1 English(EN) · Fahd Seddik, Fatemeh Fard ·

    Principled Thoughts for Latent Recursive LLM Systems

    arXiv:2609.36159v1 Announce Type: new Abstract: Large language models can reason in continuous space instead of decoded text, by recurring on their own hidden states or by passing those states between agents, while training supervises only the Cross-Entropy (CE) of the final deco…

  128. Hugging Face Daily Papers TIER_1 English(EN) ·

    Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment

    Chain-of-thought (CoT) traces often serve as a proxy for how Large Language Models (LLMs) arrive at their answers. However, growing evidence shows that models' CoT often fails to reflect their internal computations and can be changed without affecting their final answers. In this…

  129. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Natasha Jaques ·

    From Solo to Social Learning: Characterizing Recursive Social Improvement in LLMs

    Large language models (LLMs) can now improve themselves by revising the instructions they follow, and LLM agents are increasingly orchestrated to work together on complex problems. However, self-improvement methods typically optimize one system at a time, and multi-agent framewor…

  130. Hugging Face Daily Papers TIER_1 English(EN) ·

    Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs

    As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming increasingly important for quantifying the uncertainty of their answers. Current methods for estimating the confidence of large langu…

  131. Hugging Face Daily Papers TIER_1 English(EN) ·

    Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning

    On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies this supervision to unverified student rollouts while conditioning the teacher on privileged context, typically a reference solut…

  132. Hugging Face Daily Papers TIER_1 English(EN) ·

    RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications

    We study reasoning in Rule-Governed Decision Tasks (RGDTs), where models apply external rules to case facts and justify decisions, as required in policy, contract, and compliance settings. Beyond the deductive capability emphasized by standard mathematical and logical reasoning t…

  133. Hugging Face Daily Papers TIER_1 English(EN) ·

    Principled Thoughts for Latent Recursive LLM Systems

    Large language models can reason in continuous space instead of decoded text, by recurring on their own hidden states or by passing those states between agents, while training supervises only the Cross-Entropy (CE) of the final decoded answer and does not constrain the thought. T…

  134. Hugging Face Daily Papers TIER_1 English(EN) ·

    An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning

    We study on-policy distillation (OPD) through the lens of reinforcement learning, establishing a connection between the reverse-KL objective in OPD and KL-regularized policy optimization. Building on this connection, we introduce Least-Square Policy Distillation (LSPD), an RL-ins…

  135. arXiv stat.ML TIER_1 English(EN) · Yang Ba, Michelle V. Mancenido, Rong Pan ·

    Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge

    arXiv:2610.14029v1 Announce Type: cross Abstract: LLM-as-a-judge evaluation commonly aggregates multiple judges' decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable. We propose Structured Cross-f…

  136. arXiv cs.CV TIER_1 English(EN) · Zhixi Zhu, Kristina Gligoric ·

    Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics

    arXiv:2610.00809v1 Announce Type: new Abstract: Vision-Language Models (VLMs) enable video annotation at scale, but costs accumulate quickly: processing a typical 60-second short-form video at one frame per second requires millions of tokens. To reduce costs, researchers rely on …

  137. Towards AI TIER_1 English(EN) · Divye Singh ·

    Build a Jev-Style Decision Model from an LLM, Part 4: Letting the Backbone Learn, Then Giving It…

    <h3>Build a Jev-Style Decision Model from an LLM, Part 4: Letting the Backbone Learn, Then Giving It Eyes</h3><h4><em>Letting the backbone learn with LoRA and then giving the same scoring head images to read.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1…

  138. Towards AI TIER_1 English(EN) · Divye Singh ·

    Build a Jev-Style Decision Model from an LLM, Part 3: Training the Jev Head

    <h4><em>How four vectors become probabilities: the pointer head that scores the options, how it is trained and how far a frozen backbone can take it.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Oa07qLP3X1wRCu3" /><figcaption>Photo by <a href="htt…

  139. Towards AI TIER_1 English(EN) · Divye Singh ·

    Build a Jev-Style Decision Model from an LLM, Part 2: Opening the Model

    <h4><em>What is inside a hybrid vision-language model, what that rules out and how to feed it a decision.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lkqz3tuS_wHTFiWZ" /><figcaption>Photo by <a href="https://unsplash.com/@timmossholder?utm_source=…

  140. Medium — fine-tuning tag TIER_1 English(EN) · Darsh Mamtora ·

    Fine-Tuning LLMs: What, Why, and How

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@darshmamtora2122/finetuning-why-and-what-e1d206cad872?source=rss------fine_tuning-5"><img src="https://cdn-images-1.medium.com/max/1344/1*exa2Kg3MRrCX2Jhw3sktzg.png" width="1344" /></a></p><p …

  141. dev.to — MCP tag TIER_1 English(EN) · Renato Marinho ·

    Bridging the Gap Between LLM Reasoning and High-Precision Financial Engineering

    <p>Most engineers treating LLMs as reasoning engines fall into a predictable trap: they ask the model to perform complex arithmetic or heavy-duty decision logic internally. While GPT-4o or Claude 3.5 Sonnet are excellent at structured thinking, relying on their latent ability to …

  142. Medium — MLOps tag TIER_1 English(EN) · Erwin Hermanto ·

    Building an Evaluation Framework for LLM Outputs

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@erwindev/building-an-evaluation-framework-for-llm-outputs-43e4a451065b?source=rss------mlops-5"><img src="https://cdn-images-1.medium.com/max/1024/1*ppodNvvWHDSVvUDS3E83Cw.png" width="1024" />…

  143. Medium — fine-tuning tag TIER_1 English(EN) · Joya Parveen ·

    Fine-Tuning LLMs: How Do You Actually Teach an AI Model a New Skill?

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@joyaparveen0329/fine-tuning-llms-how-do-you-actually-teach-an-ai-model-a-new-skill-2645b6b6c80c?source=rss------fine_tuning-5"><img src="https://cdn-images-1.medium.com/max/2492/1*9t33zj8VC2fx…

  144. dev.to — MCP tag TIER_1 English(EN) · Renato Marinho ·

    Beyond First-Order Reasoning: Why LLM Strategies Fail Under Pressure

    <p>Most AI-generated strategies suffer from the same terminal flaw: they are forward-only. If you ask an LLM to propose a market entry plan or a scaling architecture, it will present a linear progression of successful steps. It describes the ascent, but it never maps the cliffs.<…

  145. dev.to — MCP tag TIER_1 English(EN) · Renato Marinho ·

    Moving Beyond LLM Hallucinations in Technical Analysis via Deterministic MCP Tools

    <p>Large Language Models are notoriously bad at arithmetic. When you ask a model to interpret complex financial oscillators, it isn't performing calculus; it is predicting the most likely next token based on training data. In technical analysis—where a decimal error in a volatili…

  146. Medium — fine-tuning tag TIER_1 English(EN) · SINAPSA Infocomplex ·

    LLM fine-tuning: example of a valid dataset and a chat template incorporating a Constitution for…

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@contact_30070/llm-fine-tuning-example-of-a-valid-dataset-and-a-chat-template-incorporating-a-constitution-for-476b7d2f1518?source=rss------fine_tuning-5"><img src="https://cdn-images-1.medium.…

  147. Medium — fine-tuning tag TIER_1 English(EN) · Harthik Mallichetty ·

    From Code Review to ARC-AGI-2: Reusing the Same ML Muscle on a Harder Reasoning Problem

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@hrmallichetty/from-code-review-to-arc-agi-2-reusing-the-same-ml-muscle-on-a-harder-reasoning-problem-a4be66501d3b?source=rss------fine_tuning-5"><img src="https://cdn-images-1.medium.com/max/2…

  148. Medium — fine-tuning tag TIER_1 English(EN) · Michal Molka ·

    Fine-Tuning an LLM: Teaching a Data Warehouse Schema Without RAG

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://michalmolka.medium.com/fine-tuning-an-llm-teaching-a-data-warehouse-schema-without-rag-1122ff567c41?source=rss------fine_tuning-5"><img src="https://cdn-images-1.medium.com/max/1536/1*Owot1Ud8ISf4rerCFiU5…

  149. dev.to — LLM tag TIER_1 English(EN) · RESK ·

    How an LLM Evaluation Benchmark Produces Comparable Numbers: Inside the LFORLA Mechanism

    <h1> How an LLM Evaluation Benchmark Produces Comparable Numbers: Inside the LFORLA Mechanism </h1> <p><strong>TL;DR:</strong> A leaderboard is only useful if you can trust the numbers. LFORLA's LLM evaluation makes scores comparable by fixing the prompts, fixing the judge, scori…

  150. dev.to — LLM tag TIER_1 English(EN) · Anu Kumar ·

    LLM evaluation: evals that catch regressions, and when to trust LLM-as-a-judge

    <p>Without evals, every prompt change is a guess. You fix the example in front of you and quietly break three you aren't looking at. An eval set turns that guess into a number you can watch.</p> <h2> Start from real failures </h2> <p>The best eval cases come from real inputs, esp…

  151. dev.to — LLM tag TIER_1 English(EN) · Arthur Doss ·

    Implementing Human Approval Gates in an LLM Tool-Calling Workflow

    <p>LLM agents can select tools, generate arguments, and initiate workflows, but a valid tool call does not guarantee that an action is safe or authorized. Human approval gates solve this problem by introducing a controlled checkpoint between an agent's decision and the execution …

  152. dev.to — LLM tag TIER_1 English(EN) · Olivia White ·

    A Practical Framework for Deciding When NOT to Use an LLM

    <p>Not every software problem needs a large language model (LLM). If a task can be solved reliably using deterministic rules, SQL queries, standard APIs, search algorithms, or traditional machine learning, adding an LLM may increase cost, latency, security exposure, and maintenan…

  153. dev.to — LLM tag TIER_1 English(EN) · Thomas Poli ·

    Running LLMs Locally on a Budget: A Practical 2026 Guide

    <p>Cloud AI APIs are convenient, but they come with recurring costs, latency, and the small matter of sending your data to someone else's servers. Running a language model on your own machine solves all three — and in 2026 the tooling has matured enough to be genuinely practical …

  154. dev.to — LLM tag TIER_1 English(EN) · Damiano ·

    Let the model explain, let the code count: a local LLM helper for Swiss administrative letters

    <p>In Switzerland, a letter from a public office can start a clock. A <em>Zahlungsbefehl</em> (payment order) gives you 10 days to object. An <em>Einsprache</em> period is often 30. Miss the date and you lose a right. Many people read these letters in a language that isn't their …

  155. dev.to — LLM tag TIER_1 English(EN) · Phil IPS ·

    LLM Evaluation & Benchmarking: Metrics, Frameworks & Production Testing

    <p>Liquid syntax error: Unknown tag 'endraw'</p>

  156. dev.to — LLM tag TIER_1 English(EN) · Phil IPS ·

    How to Evaluate LLMs: A Complete Guide to LLM Evaluation

    <h1> How to Evaluate LLMs: A Complete Guide to LLM Evaluation </h1> <p>From automated metrics to human evaluation — build a robust evaluation pipeline for your LLM applications.</p> <h2> Why Evaluation Matters </h2> <p>You've built an LLM app. It works on your test cases. But wil…

  157. dev.to — LLM tag TIER_1 English(EN) · Acadify Solution ·

    A Small RAG Evaluation Harness: Test Retrieval Before Blaming the LLM

    <p>A wrong RAG answer does not automatically mean the language model failed. The system may have retrieved the wrong document, omitted the relevant passage, or supplied conflicting evidence.</p> <p>This guide builds a small <strong>retrieval-only evaluation harness</strong>. It s…

  158. dev.to — LLM tag TIER_1 English(EN) · QuantID ·

    Measuring the Real Cost of an LLM Call: Tokens, Precision, and Honest Numbers

    <h2> TL;DR </h2> <p>The price per million tokens is the sticker, not the cost. The real cost of an LLM call depends on how many tokens you actually spend, the precision the provider serves at, how much the request waits, and whether the endpoint is even up. This post is a measure…

  159. dev.to — LLM tag TIER_1 English(EN) · RESK ·

    LLM Evaluation Explained: How a Benchmark Produces Comparable Numbers

    <h2> TL;DR </h2> <p>LLM evaluation only becomes useful when every model faces the same prompts, the same fixed judge, and the same per-axis rubrics, with a public verbatim trail so anyone can re-read what a model actually answered. This article explains that mechanism step by ste…

  160. dev.to — LLM tag TIER_1 English(EN) · Ashish ·

    Making LLM ratings auditable: quotes, limitations, and "can't judge" instead of zero

    <h1> From Auditable LLMs to Auditable Vision Rehab: Why Transparency Wins </h1> <p>A recent dev.to post sparked a necessary conversation about LLM-generated ratings. The author proposed a radical ruleset: fix the evidence window, show the actual quotes, admit limitations, and pro…

  161. dev.to — LLM tag TIER_1 English(EN) · RESK ·

    How LLM Evaluation Actually Works: Inside a Benchmark That Produces Comparable Numbers

    <h2> TL;DR </h2> <p>LLM evaluation only produces comparable numbers when every model faces the same prompts, the same fixed judge, per-axis rubrics, and a public verbatim trail. This article walks through how the FreeCAD Fix benchmark on LFORLA does exactly that, and what its lea…

  162. r/LocalLLaMA TIER_1 English(EN) · /u/pmttyji ·

    [Paper] EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

    <table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1x1eb7w/paper_engramedit_decoupled_knowledge_updates_in/"> <img alt="[Paper] EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory" src="https://preview.redd.it/cv4ijqtw0euh1.png?width=140…

  163. dev.to — LLM tag TIER_1 English(EN) · siddhesh kabra ·

    LLM Evaluation: 7 Critical Methods That Actually Work

    <p><strong>Published:</strong> 2026-10-03</p> <p>LLM evaluation is how you measure whether a model's output is actually correct, rather than just fast or cheap. It is the half of quality that monitoring cannot see. I built a small evaluation script in Python, scored eight model a…

  164. dev.to — LLM tag TIER_1 Français(FR) · KAMAL KISHOR ·

    LLM Self-Critique: Improve Outputs Today

    <p>Liquid syntax error: 'raw' tag was never closed</p>

  165. dev.to — LLM tag TIER_1 English(EN) · Farhad Rahimi Klie ·

    How LLMs Work: A Deep Dive From Tokens to Intelligence

    <p>Large Language Models (LLMs) have changed how we interact with computers.</p> <p>Tools such as ChatGPT, Claude, Gemini, and many open-source models can write code, explain complex topics, summarize documents, translate languages, and generate natural-sounding conversations.</p…

  166. dev.to — LLM tag TIER_1 English(EN) · RESK ·

    LLM Evaluation: How a Benchmark Turns Raw Answers Into Comparable Numbers

    <h1> LLM Evaluation: How a Benchmark Turns Raw Answers Into Comparable Numbers </h1> <p><strong>TL;DR</strong> — LLM evaluation only produces comparable numbers when every model faces the same prompts, the same fixed judge, and the same per-axis rubrics, with a public verbatim tr…

  167. dev.to — LLM tag TIER_1 English(EN) · amrit ·

    Le Chonk Unleashed: Why Mistral’s 1‑Trillion‑Parameter Model Is Shattering LLM Benchmarks

    <h2> <strong>Mistral Large 4 (“Le Chonk”) Takes the Stage: What the Benchmarks Really Reveal</strong> </h2> <h3> A Sharper Lead </h3> <p>On <strong>October 3 2026</strong>, Mistral released a <strong>1‑trillion‑parameter</strong> model that anyone can download under an open‑weigh…

  168. dev.to — LLM tag TIER_1 English(EN) · Denis Lavrentyev ·

    Overcoming LLM Knowledge Gap: Balancing AI Tool Use with Advanced Local Model Understanding

    <h2> Introduction: The Rise of LLMs and Their Impact </h2> <p>The integration of <strong>Large Language Models (LLMs)</strong> into professional workflows has sparked a critical debate: <em>Is advanced knowledge of local LLMs essential for leveraging AI tools effectively?</em> Th…

  169. dev.to — LLM tag TIER_1 English(EN) · Filipe Martins ·

    How LLMs Work: A Journey Through One Sentence

    <h1> How LLMs Work: A Journey Through Tokens, Attention, and Transformers </h1> <p>“My favourite rock band is…”</p> <p>How does a large language model take that unfinished sentence and decide what comes next?</p> <p>You’ve probably heard that LLMs “predict the next token”. But th…

  170. dev.to — LLM tag TIER_1 English(EN) · RESK ·

    How LLM Evaluation Produces Comparable Numbers: Inside the LFORLA Reverse Engineering Benchmark

    <h2> TL;DR </h2> <p>LLM evaluation only becomes useful when every model faces the same prompts, the same fixed judge, per-axis rubrics, and a public verbatim trail. The LFORLA Reverse Engineering benchmark does exactly that: it restores C source from stripped binaries, scores wit…

  171. dev.to — LLM tag TIER_1 English(EN) · Lightning Developer ·

    Scaling Intelligence: Running LLMs Across a Seven-Board ESP32-S3 Cluster

    <p>Running large language models (LLMs) on microcontrollers has long been considered a "what if" scenario reserved for theoretical discussions. However, the recent emergence of the <a href="https://github.com/Low-Zi-Hong/ESP32s3-LLM-Cluster" rel="noopener noreferrer">ESP32s3-LLM-…

  172. dev.to — LLM tag TIER_1 English(EN) · Ayi NEDJIMI ·

    LLM Fine-tuning with LoRA: A Practical Guide for Developers

    <p>Fine-tuning a large language model on your own data used to require serious GPU budgets and weeks of infrastructure work. LoRA (Low-Rank Adaptation) changed the math: you can adapt a 7B-parameter model to a specific domain in a few hours on a single consumer GPU, without touch…

  173. Mastodon — mastodon.social TIER_1 English(EN) · [email protected] ·

    A mechanism explainer on LLM evaluation using the Team Recruitment Oracle benchmark: same prompts, a fixed judge, per-axis rubrics, and a public verbatim trail.

    A mechanism explainer on LLM evaluation using the Team Recruitment Oracle benchmark: same prompts, a fixed judge, per-axis rubrics, and a public verbatim trail. Real scores for Nemotron 3 Ultra, HY3, and our own GLM 5.2. # llm # evaluation # benchmark # ai # software # coding # d…

  174. r/ClaudeAI TIER_2 English(EN) · /u/SmirkingMan ·

    Difficult reasoning competition for LLMs - Hi-ToM

    <!-- SC_OFF --><div class="md"><p>Source: <a href="https://github.com/ying-hui-he/Hi-ToM_dataset">https://github.com/ying-hui-he/Hi-ToM_dataset</a></p> <p>I asked several LLMs to solve this Hi-ToM puzzle:</p> <p>The following story happens in chronological order. You will be give…