New research explores LLM efficiency and reasoning improvements
ByPulseAugur Editorial·[222 sources]·
Several research papers explore methods to enhance the efficiency and reliability of large language models (LLMs). Hugging Face's LFM2.5-DSpark demonstrates up to 3.2x faster inference speeds by using speculative decoding techniques. OpWeave and LayerRoute propose new frameworks for optimizing LLM serving across heterogeneous hardware and enabling adaptive layer-skipping, respectively. Other research investigates LLM reasoning and calibration, with ZebraArena providing a diagnostic environment for tool-augmented LLMs and Efficiency Hallucination formalizing the risk of LLMs issuing non-functional code mutations.
AI
IMPACT
These advancements in LLM efficiency and reasoning could lead to faster, more reliable AI applications and better diagnostic tools for evaluating model behavior.
RANK_REASON
Multiple arXiv papers detailing novel research into LLM efficiency and reasoning.
arXiv:2609.19515v1 Announce Type: new Abstract: Verifier-based selection improves LLM performance by generating multiple candidate solutions and using a verifier to select the most promising one. However, existing methods typically treat verification only as a ranking step and di…
arXiv cs.AI
TIER_1English(EN)·Chao Wang (Independent Researcher)·
arXiv:2609.19182v1 Announce Type: new Abstract: Benchmarks are central to how progress in large language models (LLMs) is assessed and communicated. Yet model rankings alone reveal little about how evaluation requirements themselves are changing. The expanding variety of benchmar…
arXiv:2602.04620v3 Announce Type: replace Abstract: GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as global import…
arXiv:2609.19657v1 Announce Type: cross Abstract: Repeated prompt prefixes are increasingly common in LLM serving workloads, appearing in system prompts, templated retrieval-augmented generation pipelines, agent frameworks, and multi-turn conversations. Modern inference runtimes …
arXiv cs.LG
TIER_1English(EN)·Mobina Kashaniyan, Ali Jannesari·
arXiv:2609.19499v1 Announce Type: new Abstract: Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. Howe…
arXiv cs.LG
TIER_1English(EN)·Tejus Gupta, Efe Mert Karag\"ozl\"u, Rohit Sonker, Barnab\'as P\'oczos, Jeff Schnieder·
arXiv:2609.19437v1 Announce Type: new Abstract: Bayesian Optimization (BO) is widely used for optimizing expensive black-box functions, yet many real-world optimization problems contain substantially richer information than function evaluations alone. Examples include training cu…
arXiv:2605.18630v2 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly deployed as scientific AI as- sistants, and a growing body of benchmarks evaluates their capabilities across knowledge retrieval, reasoning, code generation, and tool use. These eval…
arXiv:2512.13142v5 Announce Type: replace Abstract: People use LLMs for reproductive-health questions, including abortion-related support. A response can sound supportive while answers reinforce harmful assumptions: judgment is likely, secrecy is safer, and support is limited. We…
arXiv cs.AI
TIER_1English(EN)·Xin Chen, Gil Kur, Alexander Shevchenko, Andreas Krause·
arXiv:2609.20129v1 Announce Type: cross Abstract: Deployment-time safety methods for large language models (LLMs) are predominantly supervised and assume access to unsafe training data. Nevertheless, new attacks and harm categories regularly arise, not captured by models trained …
arXiv cs.AI
TIER_1English(EN)·Pyrros Koussios, Benjamin J\"ager, John Hua Yao, Ajay Sridhar, Violet Xiang, Chenhao Li·
arXiv:2609.19883v1 Announce Type: cross Abstract: Characterizing LLM reasoning remains an open challenge, as many existing benchmarks isolate specific reasoning skills, rely on external knowledge, or are costly to extend. We introduce PetriBench, a compact, fully self-contained, …
Deployment-time safety methods for large language models (LLMs) are predominantly supervised and assume access to unsafe training data. Nevertheless, new attacks and harm categories regularly arise, not captured by models trained in such a supervised fashion. An alternative appro…
arXiv cs.AI
TIER_1English(EN)·Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin·
arXiv:2609.19144v1 Announce Type: cross Abstract: Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to …
arXiv:2609.18723v1 Announce Type: new Abstract: We introduce Mahalanobis-Ensemble Decoding (ME-Decoding), a novel Large Language Model (LLM) decoding framework that frames candidate token selection as ensemble pruning. Existing selection strategies rely predominantly on scalar pr…
arXiv:2609.17573v1 Announce Type: cross Abstract: Diffusion large language models (dLLMs) are emerging as a promising generative paradigm that complements autoregressive decoding. In long-context settings, KV cache bloat and offloading transfer overhead have become primary bottle…
arXiv:2609.19072v1 Announce Type: new Abstract: Large language models are increasingly used for content moderation, but most evaluations still report aggregate accuracy on individual benchmarks. We introduce Safety-Flag, which places seven widely used safety benchmarks (BeaverTai…
arXiv cs.AI
TIER_1English(EN)·Jinli Hu, Ross M. Clarke, Yichuan Zhang, Jos\'e Miguel Hern\'andez-Lobato·
arXiv:2609.18842v1 Announce Type: new Abstract: The scaling laws hold that a language model grows more capable with more parameters and more training data, and Mixture-of-Experts (MoE) architectures have ridden these laws to remarkable results, activating only a fraction of an en…
arXiv cs.AI
TIER_1English(EN)·Kirill Skobelev, Eric Fithian, X. Y. Han·
arXiv:2609.16454v1 Announce Type: new Abstract: Recent work by Doshi and Hauser (2024), Bisbee et al. (2024), and Xie et al. (2026) raises concerns that outputs from large language models (LLMs) tend to be under-diverse: they repeat or resemble one another more often than respons…
arXiv cs.LG
TIER_1English(EN)·Sebastian Zhao, Minseo Kim, Coleman Hooper, Luca Manolache, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami·
arXiv:2609.16161v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown impressive capabilities across a range of natural language processing tasks, and LLM inference has emerged as a critical workload for enabling downstream applications. The demands of serving L…
arXiv cs.CL
TIER_1English(EN)·Aashiq Muhamed, Mona T. Diab, Virginia Smith·
arXiv:2609.16204v1 Announce Type: cross Abstract: Safety guardrails in open-weight language models can be readily bypassed using Refusal Feature Ablation (RFA), a technique that identifies and projects out a linear refusal direction from the residual stream, often achieving a hig…
arXiv cs.AI
TIER_1English(EN)·Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude·
arXiv:2604.06723v2 Announce Type: replace-cross Abstract: In today's AI-assisted software engineering landscape, developers increasingly depend on LLMs that are highly capable, yet inherently imperfect. The tendency of these models to produce incorrect outputs can reduce develope…
arXiv cs.AI
TIER_1English(EN)·Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu·
arXiv:2604.05012v2 Announce Type: replace-cross Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value vectors at each layer. These caches are essential to minimize redundant computat…
arXiv cs.AI
TIER_1English(EN)·Qihu Xie, Ziwei Li, Yi Kang·
arXiv:2609.17008v1 Announce Type: new Abstract: Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights are transferred across memory hierarchies during autoregressive decoding. In th…
arXiv cs.AI
TIER_1English(EN)·Bernie Boscoe, Srinath Saikrishnan, Vikram Seenivasan, Jack Stark, Andrew Lizarraga, Morgan Himes, Jonathan Soriano, PJ Allen, Tuan Do·
arXiv:2609.16519v1 Announce Type: new Abstract: Scientific research increasingly relies on large, heterogeneous data sources, motivating interest in retrieval-augmented generation (RAG) systems that provide natural language access to scientific knowledge and research workflows. R…
arXiv cs.AI
TIER_1English(EN)·Evangelos Georganas, Alexander Heinecke, Pradeep Dubey·
arXiv:2609.16338v1 Announce Type: new Abstract: Ternary Large Language Models (LLM) store every weight as one of three symbols $\{-1,0,+1\}$, so the cost of a ternary model is conventionally referenced to the information-theoretic $\log_2 3 \approx 1.585$ bits per weight. The pre…
arXiv cs.AI
TIER_1English(EN)·Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly·
arXiv:2609.16206v1 Announce Type: new Abstract: Disaggregated LLM serving places compute heavy prefill and memory heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast, but routing still determines which instances handle …
Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with sma…
Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. However, N tells us how many candidates are generate…
arXiv:2509.24975v4 Announce Type: replace-cross Abstract: Diffusion large language models (dLLMs) enable parallel generation and are promising for unit test generation (UTG), where efficient and large-scale automated testing is essential in software development. Despite this adva…
arXiv cs.CL
TIER_1English(EN)·Zo\"e Prins, Samuele Punzo, Frank Wildenburg, Giovanni Cin\`a, Sandro Pezzelle·
arXiv:2603.29396v2 Announce Type: replace Abstract: Standard evaluations of Large language models (LLMs) focus on task performance, offering limited insight into whether correct behavior reflects appropriate underlying mechanisms and risking confirmation bias. We introduce a simp…
arXiv cs.CL
TIER_1English(EN)·Hyeongcheol Park, Sumin In, Suyeon Myeong, Hogun Park, Sangmin Kim, Moonhyun Lee, Daekyeong Park, Sangpil Kim·
arXiv:2609.15022v1 Announce Type: new Abstract: Defense is a knowledge-intensive domain that requires precise understanding of specialized terminology, doctrinal concepts, operational procedures, and evolving military events. Although recent work has explored language technologie…
arXiv cs.AI
TIER_1English(EN)·Angelos Poulis, Mark Crovella, Evimaria Terzi·
arXiv:2604.03754v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have been shown to encode truth of statements in their activation space along a linear truth direction. Previous studies have argued that these directions are universal in certain aspects, whil…
arXiv cs.AI
TIER_1English(EN)·Shaghayegh Sadeghi, Stephen L. Smith, David C. Del Rey Fern'andez·
arXiv:2609.13945v1 Announce Type: new Abstract: Natural-language descriptions of optimization problems may be incomplete or vague about numerical information that a solver requires, including costs, capacities, demands, bounds, and penalties. A language model can translate the de…
arXiv:2609.13592v1 Announce Type: cross Abstract: GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-bandwidth memory (HBM) and a secondary tier of host memory connected via CPU-to-G…
arXiv cs.AI
TIER_1English(EN)·Wanjia Zhao, Ludwig Schmidt, Yejin Choi, James Zou, Vidhisha Balachandran, Lingjiao Chen·
arXiv:2603.18614v2 Announce Type: replace Abstract: Tool-augmented large language models (LLMs) must tightly couple multi-step reasoning with external actions, yet existing benchmarks often confound this interplay with complex environment dynamics, memorized knowledge or dataset …
arXiv cs.AI
TIER_1English(EN)·Sarah Wilson, Gail Kaiser, Patrick Musau·
arXiv:2609.14839v1 Announce Type: cross Abstract: The integration of Large Language Models (LLMs) into automated code optimization introduces a critical reliability risk we term the Efficiency Hallucination: an LLM's tendency to issue non-functional mutations with unsubstantiated…
arXiv:2609.14237v1 Announce Type: cross Abstract: LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve ha…
arXiv:2609.13682v1 Announce Type: cross Abstract: We introduce LayerRoute, a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer hard-gated routing (trained via a straight-through estimator) with joint LoRA fine-tuning. LayerRoute augments e…
arXiv:2609.12002v1 Announce Type: new Abstract: LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings;…
arXiv cs.CL
TIER_1English(EN)·Foad Namjoo, Remy Ogasawara, Amirali Abdullah, Cullen Anderson, Narmeen Fatimah Oozeer, Jeff M. Phillips·
arXiv:2609.13003v1 Announce Type: new Abstract: Binary-choice truth benchmarks ask models to choose between a correct and an incorrect answer, but if the two answers differ systematically in surface-level features, models can exceed chance without performing the intended reasonin…
arXiv:2507.01663v2 Announce Type: replace Abstract: Reinforcement learning (RL) has become a pivotal technology in the post-training phase of large language models (LLMs). Traditional task-collocated RL frameworks suffer from significant scalability bottlenecks, while task-separa…
Safety guardrails in open-weight language models can be readily bypassed using Refusal Feature Ablation (RFA), a technique that identifies and projects out a linear refusal direction from the residual stream, often achieving a high attack success rate (ASR) while preserving model…
arXiv:2609.11393v1 Announce Type: new Abstract: Test-time adaptation has emerged as a lightweight alternative to costly post-training for improving the reasoning capabilities of Large Language Models (LLMs) on downstream tasks. Predictive entropy provides a model-derived signal f…
arXiv:2609.11180v1 Announce Type: new Abstract: Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBen…
arXiv cs.CL
TIER_1English(EN)·Adithiyan Rajan Indira Saravanan, Kathleen C. Fraser·
arXiv:2609.11758v1 Announce Type: new Abstract: Allowing large language models (LLMs) to retrieve information from a set of trusted documents can increase reliability and reduce hallucination. However, recent work has demonstrated that retrieval-augmented generation (RAG) can hav…
arXiv:2604.00280v2 Announce Type: replace-cross Abstract: Formal specifications play a central role in ensuring software reliability, yet automatically synthesizing high-quality specifications remains difficult and often requires domain expertise. Recent work has applied large la…
arXiv cs.AI
TIER_1English(EN)·Marek Jeli\'nski, Jan Dubi\'nski, Maciej Chrabaszcz, Sebastian Cygert·
arXiv:2609.10060v1 Announce Type: new Abstract: Existing bias auditing methods typically rely on model outputs, requiring costly benchmarks or judge models and potentially missing internal shifts that never appear in generated text. We propose a reference-based method that audits…
arXiv cs.IR (Information Retrieval)
TIER_1English(EN)·Kathleen C. Fraser·
Allowing large language models (LLMs) to retrieve information from a set of trusted documents can increase reliability and reduce hallucination. However, recent work has demonstrated that retrieval-augmented generation (RAG) can have unintended side effects on the overall safety …
Allowing large language models (LLMs) to retrieve information from a set of trusted documents can increase reliability and reduce hallucination. However, recent work has demonstrated that retrieval-augmented generation (RAG) can have unintended side effects on the overall safety …
Test-time adaptation has emerged as a lightweight alternative to costly post-training for improving the reasoning capabilities of Large Language Models (LLMs) on downstream tasks. Predictive entropy provides a model-derived signal for such adaptation, guiding models toward higher…
arXiv cs.LG
TIER_1English(EN)·Antonin Poch\'e, Fanny Jourdan, Nils Feldhus, Qianli Wang, Jing Yang, Simon Ostermann, Nicholas Asher, Philippe Muller, Vera Schmitt·
arXiv:2609.08585v1 Announce Type: cross Abstract: Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model's outputs. Since human evaluation is costly, automated simulatability replaces human expl…
arXiv:2609.06993v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly generate Markdown that is consumed by renderers, agents, code extractors, and structured downstream pipelines. Yet existing evaluations often conflate content quality with format adherence…
arXiv:2609.05565v1 Announce Type: cross Abstract: Large language model (LLM) sustainability is increasingly a serving-systems problem, not only a training problem. In production, energy and carbon impact depend on more than model size: workload shape, batching, key-value (KV) cac…
arXiv:2609.06557v1 Announce Type: new Abstract: Large language models (LLMs) are often considered fragile under aggressive sparsification, and maintaining reliable performance typically requires sticking to moderate sparsity levels. However, recent studies suggest that LLMs are m…
arXiv cs.LG
TIER_1English(EN)·Saurav Bhandari, Benjamin Wade·
arXiv:2609.06473v1 Announce Type: new Abstract: Inference-time activation steering enables behavioral control of large language models without parameter modification, while post-training quantization reduces memory and compute costs for deployment. Despite their growing convergen…
arXiv cs.LG
TIER_1English(EN)·Anna-Christina Glock, Thomas H\"utter, Johannes F\"urnkranz, Wolfram W\"o{\ss}, Christine Dominka-Kiss, Lisa Ehrlinger·
arXiv:2609.06053v1 Announce Type: new Abstract: The validation of data, such as customer and employee data, is an important task in many organizations. Errors in data can have severe consequences. For example, a wrong drug unit in a patient record can lead to life-threatening med…
arXiv:2609.05658v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly being explored for automating SystemVerilog Assertion (SVA) generation, yet most evaluations report correctness on a single syntactic representation of an input. Such point accuracy does…
arXiv:2609.10397v1 Announce Type: cross Abstract: Exception Related Code (ERC), which includes throw statements, conditions (if statements) that guard those throw statements, and try/catch blocks, is an essential component of software systems, allowing developers to detect and ha…
arXiv cs.CL
TIER_1English(EN)·Xietao Wang-Lin, Anton Isopoussu, Louis Mahon·
arXiv:2609.10123v1 Announce Type: cross Abstract: Large language models (LLMs) have become ubiquitous in software development, with LLM-based automated program repair tools increasingly used during code review. In this report, we explore the iterative blind use of LLMs as bug-fix…
arXiv:2609.09815v1 Announce Type: cross Abstract: Compound LLM systems often solve a coordination problem by adding a higher-level LLM. The resulting meta-agent reads workers' outputs, writes the final answer, allocates later calls, and decides when to stop. It is expressive, but…
arXiv:2603.00039v2 Announce Type: replace-cross Abstract: LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality.…
arXiv cs.AI
TIER_1Norsk(NO)·Yuchen Yang, Wenze Lin, Enhao Huang, Zhixuan Chu, Hongbin Zhou, Lan Tao, Yiming Li, Zhan Qin, Kui Ren·
arXiv:2602.14536v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have seen remarkable advancements, achieving state-of-the-art results in diverse applications. Fine-tuning, an important step for adapting LLMs to specific downstream tasks, typically involves …
arXiv cs.AI
TIER_1English(EN)·Maysam Khatib, Moysis Symeonides, Demetris Trihinas, George Pallis, Marios D. Dikaiakos·
arXiv:2609.08307v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires balancing quality, latency, model footprint, and energy. This paper presents a contro…
arXiv cs.AI
TIER_1English(EN)·Shahzeb Qamar, Lorenz Sparrenberg, Christian Bauckhage, Baha Rababah, Carson Leung, Murat Kantarcioglu, Cuneyt Gurcan Akcora, Rafet Sifa·
arXiv:2609.07664v2 Announce Type: cross Abstract: Deployment of Large Language Models (LLMs) on memory-constrained edge devices relies heavily on aggressive post-training quantization. However, evaluating these models is largely based on zero-shot task accuracy, which depends sol…
arXiv:2609.06229v1 Announce Type: cross Abstract: Vulnerability discovery is becoming an important ability of large language model (LLM) agents: agents that silently miss real defects leave critical software exposed. Rigorously measuring this ability is therefore urgent, but exis…
arXiv cs.AI
TIER_1English(EN)·Ahin Lee, Sehyun Yun, Joonha Park, Taesik Gong·
arXiv:2609.06072v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-wise design fragments adaptation in three ways: capacity is split across narrow lo…
arXiv cs.AI
TIER_1English(EN)·Jie Ruan, Inderjeet Nair, Amy Liu, Muhammad Khalifa, Yusheng Zhou, Lu Wang·
arXiv:2609.08126v1 Announce Type: new Abstract: We study scheming in LLM agents, in which agents covertly pursue misaligned goals. Our focus is to understand how scheming arises from the interaction of key factors, such as instrumental goals, environmental affordances, oversight …
arXiv:2609.05512v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) impose substantial energy costs during deployment, yet current compression methods apply uniform quantization across all components, risking damage to critical reasoning circuits. We present a reasoning…
Large language models (LLMs) have become ubiquitous in software development, with LLM-based automated program repair tools increasingly used during code review. In this report, we explore the iterative blind use of LLMs as bug-fixers. Across multiple models and repair environment…
Compound LLM systems often solve a coordination problem by adding a higher-level LLM. The resulting meta-agent reads workers' outputs, writes the final answer, allocates later calls, and decides when to stop. It is expressive, but it also concentrates three control decisions in a…
The study proposes a reference-based hidden-state auditing method that measures representational bias shift across model variants to detect internal bias changes with minimal compute.
Researchers introduce a 400-scenario benchmark and evidence-based monitor to study how instrumental goals, oversight, and strategic hints drive covert misaligned behavior in LLM agents.
arXiv cs.AI
TIER_1English(EN)·Qinglan Luo, S M A Nahian, John Guttag, S. Mazdak Abulnaga, Katie Matton·
arXiv:2609.04343v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for consequential decisions, making their explanations an important tool for auditing model behavior. Unfortunately, these explanations can be unfaithful, failing to reflect the act…
arXiv cs.AI
TIER_1English(EN)·Mostafa Elhoushi, Alex Pretko, Nolan Dey, Bin Claire Zhang, Gavia Gray, Gurpreet Gosal, Abdulrahman Mahmoud, Shane Bergsma, Joel Hestness·
arXiv:2609.05275v1 Announce Type: new Abstract: Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, drop…
arXiv cs.AI
TIER_1English(EN)·Jiahe Geng, Jinpeng Wang, Kun Yuan·
arXiv:2609.04915v1 Announce Type: new Abstract: Many long-horizon LLM deployments face tight prompt budgets: latency, cost, and context limits make full-context prompting impractical as interaction length grows. The key question is then not raw recall alone, but which memory desi…
arXiv:2609.04748v1 Announce Type: cross Abstract: Prefix caching, in which a serving engine reuses the key and value tensors of a shared prompt prefix across requests, is enabled by default in the major open-source stacks and treated as a transparent optimization. We measure what…
arXiv cs.AI
TIER_1English(EN)·Gnaneswar Villuri, Hashmath Shaik, Alex Doboli·
arXiv:2609.04570v1 Announce Type: cross Abstract: LLM-based code generation fails when correctness depends on execution-dependent coupling: the meaning of one routine is defined by the runtime behavior of another, a relationship that cannot be resolved from textual descriptions a…
arXiv cs.LG
TIER_1English(EN)·Renyuan Liu (Richard), Yuyang Leng (Richard), Kaiyan Liu (Richard), Yuzhou Zhong (Richard), Shaohan Hu (Richard), Chun-Fu (Richard), Chen, Peijun Zhao, Heechul Yun, Shuochao Yao·
arXiv:2609.03079v1 Announce Type: new Abstract: On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload weig…
arXiv:2609.03844v1 Announce Type: cross Abstract: We introduce Stateless Bernoulli Watermarking (SBW), a new statistical watermark for Large Language Models that determines green list membership through independent per-token Bernoulli trials. Unlike KGW's vocabulary permutation o…
arXiv:2601.10187v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have achieved remarkable strides in multilingual translation but are hindered by a systemic cross-lingual verbosity bias, rendering them unsuitable for strict time-constrained tasks like subtit…
arXiv:2609.03894v1 Announce Type: new Abstract: Large language models (LLMs) have become ubiquitous tools for code generation and editing. However, development teams often use multiple LLM assistants. Different developers may prefer different models, and individual developers may…
arXiv:2609.03494v1 Announce Type: new Abstract: Long-output reasoning has made the key--value (KV) cache a critical memory bottleneck for efficient LLM serving. Existing KV compression methods usually rely on a predefined per-request budget and adjust only which KV states are ret…
arXiv cs.LG
TIER_1English(EN)·Bigyan Ghimire, Jon C. Calhoun·
arXiv:2609.03151v1 Announce Type: cross Abstract: Long-context reasoning for large language models (LLMs) is becoming increasingly important, but training over long sequences remains challenging due to massive memory and communication requirements. Sequence parallelism has emerge…
arXiv cs.LG
TIER_1English(EN)·Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham, Jonathan Geuter, Chaitanya Dwivedi, Varad Pimpalkhute, Yash Akhauri, Alexander Moreno, Mikhail Yurochkin, Zhenting Wang, Mostafa Elhoushi, Nolan Dey, Shane Bergsma, Joel Hestness, John Thickstun, Eric Xing, …·
arXiv:2609.04010v1 Announce Type: new Abstract: Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation. To overcome this bottleneck, we introduce diffusion-augmented…
Layer dropout improves large language model training efficiency and enables faster inference via early exit and speculative decoding without sacrificing accuracy.
Large language models (LLMs) are increasingly used for consequential decisions, making their explanations an important tool for auditing model behavior. Unfortunately, these explanations can be unfaithful, failing to reflect the actual reasoning underlying the model's decisions. …
Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation. To overcome this bottleneck, we introduce diffusion-augmented LLMs, a new class of models that defines an AR …
Long-output reasoning has made the key--value (KV) cache a critical memory bottleneck for efficient LLM serving. Existing KV compression methods usually rely on a predefined per-request budget and adjust only which KV states are retained, leaving the total capacity fixed througho…
Diffusion-augmented autoregressive language models use parallel token sampling via distilled diffusion weights and a specialized sampler to accelerate inference without quality loss or draft models.
arXiv cs.AI
TIER_1English(EN)·Jingtan Wang, Arun Verma, Xiaoqiang Lin, Zhengyuan Liu, Nancy F. Chen, Daniela Rus, Bryan Kian Hsiang Low·
arXiv:2609.01573v1 Announce Type: cross Abstract: How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open problem. Existing work characterizes only broad trends (e.g., SFT dominates in l…
arXiv cs.AI
TIER_1English(EN)·Yipin Guo, Arun M George, Jie Fu, Tareq Mahmoud, Sixue Xing, Siddharth Joshi·
arXiv:2609.00224v1 Announce Type: cross Abstract: Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existing PTQ methods often fail to generalize across models and suffer severe accuracy …
arXiv cs.AI
TIER_1English(EN)·Dheeraj Mohandas Pai, Lu Xian·
arXiv:2609.00012v1 Announce Type: new Abstract: Long-horizon tasks remain uncommon in large language model (LLM) evaluation, and for a reason: when each step depends on the last, per-step accuracy that looks excellent in isolation decays catastrophically, as errors cascade and th…
arXiv:2510.01030v2 Announce Type: replace Abstract: The human ability to translate diverse perceptual and linguistic inputs into structured behavior has been thought to rest on learning robust representations of concepts. The rapid advancement of transformer-based large language …
arXiv:2505.22156v4 Announce Type: replace Abstract: Although existing model editing methods perform well in recalling exact edit facts, they often struggle in complex scenarios that require deeper semantic understanding rather than mere knowledge regurgitation. Leveraging the str…
arXiv:2609.00363v1 Announce Type: new Abstract: Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance. We measure what such a suite can detect. Injecting nine faults into a reference INT8 pipeline over 8,232 layer--fault--regime ce…
arXiv cs.LG
TIER_1English(EN)·Wentao Ye, Zhanming Shen, Zhiqing Xiao, Yao Ding, Haobo Wang, Gang Chen·
arXiv:2609.00762v1 Announce Type: new Abstract: Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential. We study this choice through fr…
Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential. We study this choice through frozen-core adaptation: a calibration pass fixes l…
arXiv cs.CL
TIER_1English(EN)·Daeha Lee, Do-Hyung Kim, Jae-Hong Kim·
arXiv:2608.28911v1 Announce Type: cross Abstract: The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length. We show that uniform KV quantization on a fractional-bit grid does not degrade …
arXiv:2608.30076v1 Announce Type: new Abstract: Single-GPU deployment of 70B-parameter language models on an NVIDIA GPU is constrained by device memory, long-context throughput, and engineering integration cost. We cast single-GPU inference as a budget-aware design problem over t…
arXiv:2512.17452v4 Announce Type: replace-cross Abstract: Long-context LLM inference is bottlenecked by the quadratic attention complexity and linear Key-Value (KV) cache growth. Prior approaches mitigate this via post-hoc selection or eviction but overlook the root inefficiency:…
arXiv:2512.16391v2 Announce Type: replace-cross Abstract: Attention is the dominant source of latency during long-context LLM inference, an increasingly popular workload with reasoning models and RAG. We propose Kascade, a training-free sparse attention method that leverages know…
arXiv:2608.28726v1 Announce Type: new Abstract: The remarkable performance of multimodal large language models (MLLMs) comes at the cost of substantial computational overhead, posing significant challenges to real-time deployment and cost effectiveness. Existing model routing app…
arXiv:2608.28846v1 Announce Type: cross Abstract: Layer-skipping methods for efficient LLM inference decide, at some granularity, which transformer layers to execute for a given input. We present a rigor-matched, three-seed audit of two periodic-step, search-based methods that ma…
arXiv:2608.30295v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challenging due to the substantial memory requirements and inference latency. In this wor…
arXiv cs.AI
TIER_1English(EN)·Jin Gan, Xin Li, Jun Luo·
arXiv:2608.30632v1 Announce Type: cross Abstract: Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models' (LLMs) reasoning abilities, demonstrating remarkable effectiveness across…
arXiv cs.AI
TIER_1English(EN)·Dongseok Lee, Jimyung Hong, Dongyoung Kim, Jaehyung Kim·
arXiv:2506.17251v3 Announce Type: replace-cross Abstract: Although large language models (LLMs) have achieved remarkable performance, the inherent stochasticity of their reasoning processes and varying conclusions present significant challenges. Majority voting or Best-of-N with …
arXiv cs.CL
TIER_1English(EN)·Yasmin Moslem, John D. Kelleher·
arXiv:2603.04445v3 Announce Type: replace-cross Abstract: The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent model selection at inference time. While smaller models suffice for routine queries…
arXiv:2604.02830v3 Announce Type: replace Abstract: Detecting whether a model's internal knowledge is sufficient to correctly answer a given question is a fundamental challenge in deploying responsible LLMs. In addition to verbalising the confidence by LLM self-report, more recen…
Large language models (LLMs) rely on static pretraining corpora, causing their knowledge to become outdated over time. Existing approaches for evaluating knowledge edits either suffer from rapid contamination or rely on counterfactual edits that conflict with rigid existing knowl…
arXiv cs.LG
TIER_1English(EN)·Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton, John Paul Shen·
arXiv:2608.28044v1 Announce Type: cross Abstract: Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can decreas…
arXiv:2608.27460v1 Announce Type: new Abstract: Large output embedding matrices create a significant memory bandwidth bottleneck during autoregressive decoding, especially for compact LLMs with large multilingual vocabularies. We reformulate the output projection followed by top-…
arXiv:2608.28113v1 Announce Type: new Abstract: The NVIDIA Blackwell architecture, with native support for the ultra-fine-grained NVFP4 format, opens new opportunities for accelerating large language model (LLM) inference. NVFP4's micro-block design, such as a group size of 16, o…
arXiv:2606.22179v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed as black-box classifiers in pipelines that automate confident decisions and route uncertain ones to human review. Such selective prediction needs a confidence score that an …
arXiv:2608.26120v1 Announce Type: new Abstract: Large Language Models (LLMs) are probabilistic models, typically defined by an autoregressive factorization. While recent work has begun to study richer target distributions beyond the base model, the sampling strategies remain high…
arXiv cs.AI
TIER_1English(EN)·Thi Thanh Nhan Nguyen, Mai Khoi Tieu, Michael A. Riegler, P{\aa}l Halvorsen, Thu Nguyen·
arXiv:2608.26186v1 Announce Type: cross Abstract: This study examines how prompt and response language influence the behavior of large language models. Using five models, we evaluated answers to 68 non translation questions across four language conditions: English to English, Eng…
arXiv cs.LG
TIER_1English(EN)·Gongwei Lee, Ji Liu, Juncheng Jia, Ji Wu·
arXiv:2608.24945v1 Announce Type: new Abstract: Recent years have witnessed remarkable achievements of Large Language Models (LLMs) in multiple domains, while the excessive resource requirements of LLMs hinder the deployment on resource-constrained devices. Although model quantiz…
arXiv:2608.25894v1 Announce Type: new Abstract: Large language models (LLMs) frequently produce confident yet factually incorrect responses when user inputs contain misleading premises, a phenomenon we attribute to fact perturbations in the input. Existing approaches to hallucina…
arXiv:2608.25824v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as evaluators to assess output quality and preference alignment, yet providing reliable guarantees of agreement with human judgments remains challenging. Recent work introduces conf…
arXiv cs.CL
TIER_1English(EN)·Lukas Edman, Daryna Dementieva, Alexander Fraser·
arXiv:2608.25605v1 Announce Type: new Abstract: Large language models (LLMs) demonstrate impressive performance across a wide range of general NLP tasks; however, their effectiveness in sensitive domains, such as hate speech detection, remains less clear. Prior studies comparing …
arXiv cs.CL
TIER_1English(EN)·Nayoung Kim, Mickey Mancenido, Huan Liu·
arXiv:2608.25379v1 Announce Type: new Abstract: Chain-of-thought prompting can expose and amplify demographic stereotypes within an LLM's intermediate reasoning and create a failure mode that final-answer debiasing alone cannot address. Mitigating such bias during generation pres…
arXiv:2608.25291v1 Announce Type: new Abstract: Symbolic regression (SR) seeks to discover parsimonious mathematical laws from observational data, yet conventional approaches often struggle with the vast combinatorial search space of physically meaningful expressions. We present …
arXiv:2505.23022v2 Announce Type: replace Abstract: Large Language Model (LLM) serving increasingly underpins online Web services such as conversational agents, Web search, and programming assistants, where requests carry heterogeneous Service Level Objectives (SLOs) such as Time…
arXiv cs.LG
TIER_1English(EN)·Vedant Shah, Johan Obando-Ceron, Vineet Jain, Brian Bartoldson, Bhavya Kailkhura, Sarthak Mittal, Glen Berseth, Pablo Samuel Castro, Yoshua Bengio, Esmeralda S. Whitammer, Moksh Jain, Siddarth Venkatraman, Aaron Courville·
arXiv:2512.21852v4 Announce Type: replace Abstract: The reasoning performance of large language models (LLMs) can be substantially improved by training them with reinforcement learning (RL). The RL objective for LLM training involves a regularization term, which is the reverse Ku…
arXiv cs.LG
TIER_1English(EN)·Tilman Hinnerichs, Sebastijan Dumancic, Neil Yorke-Smith·
arXiv:2608.25657v1 Announce Type: cross Abstract: Large language models (LLMs) excel at programming, but not when the task fixes the target language: prompted with a grammar rare in their training data, their programs usually break the grammar or fail the given specification. Enu…
arXiv cs.LG
TIER_1English(EN)·Suchit Gupte, Xueru Zhang, Mohammad Mahdi Khalili·
arXiv:2608.25941v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models (LLMs), yet their reliability under post-hoc model compression remains poorly understood. We present a systematic study of…
Large language models (LLMs) frequently produce confident yet factually incorrect responses when user inputs contain misleading premises, a phenomenon we attribute to fact perturbations in the input. Existing approaches to hallucination mitigation typically assume reliable user i…
Large language models (LLMs) demonstrate impressive performance across a wide range of general NLP tasks; however, their effectiveness in sensitive domains, such as hate speech detection, remains less clear. Prior studies comparing prompted LLMs with state-of-the-art encoder-base…
arXiv cs.AI
TIER_1English(EN)·Anh Ho, Thanh Le-Cong, Bach Le, Christine Rizkallah·
arXiv:2506.13182v3 Announce Type: replace-cross Abstract: [...] Since then, various APR approaches, especially those leveraging the power of large language models (LLMs), have been rapidly developed to fix general software bugs. Unfortunately, the effectiveness of these advanced …
arXiv:2512.13979v2 Announce Type: replace Abstract: Large reasoning models achieve strong performance on diverse tasks by producing extended chains of thought. Self-reflection, the ability to review and revise prior reasoning steps, is widely regarded as a key contributor to this…
arXiv:2608.23568v1 Announce Type: new Abstract: Memory and RAG evaluations often treat the answering model's input as an implementation detail, even though systems may render the same history as a memory entry, summary, typed record, or raw excerpt. We introduce RENDER, a benchma…
Edge LLM deployment is shaped by more than model size and precision: inference backend, hardware platform, memory traffic, and power management all affect latency and efficiency. We present Hydra, a common-schema, phase-aware workload characterization framework for LLM inference …
arXiv:2608.19147v1 Announce Type: cross Abstract: Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, and they spend considerable time idle. That is not enough memory to fit a large model such as a 70B-parameter LLM. We show that a handful of …
arXiv:2608.18591v1 Announce Type: new Abstract: Uniformly allocating inference reasoning budgets to LLMs is expensive and prone to over-thinking penalties; especially in document tasks where visual layouts drive complexity. To address this, we introduce BudgetDoc, the first multi…
arXiv:2608.17336v1 Announce Type: new Abstract: Long-context prefill in large language models (LLMs) incurs substantial computation and memory traffic because dense self-attention computes quadratic query-key scores. Existing methods either use a uniform low-precision path or sel…
arXiv:2608.15602v1 Announce Type: cross Abstract: While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the f…
arXiv:2608.16477v1 Announce Type: new Abstract: AI-RAN brings large language model (LLM) serving close to mobile users, but cellular handover can separate an active request from its inference state: the user attaches to a target base station (gNB) while the large and growing key-…
arXiv:2608.15665v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making convergence unstable and highly sensitive to learning ra…
arXiv cs.CL
TIER_1English(EN)·Ozgur Guldogan, Jackson Kunde, Kangwook Lee, Ramtin Pedarsani·
arXiv:2412.04504v2 Announce Type: replace Abstract: As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the …
TileMix routes attention score tiles to mixed FP16 or INT8 precision within fused dense attention, recovering long-context accuracy while improving prefill throughput without retraining.
AI-RAN brings large language model (LLM) serving close to mobile users, but cellular handover can separate an active request from its inference state: the user attaches to a target base station (gNB) while the large and growing key-value (KV) cache remains at the source. Retainin…
arXiv:2608.13756v1 Announce Type: new Abstract: Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable. We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration …
arXiv:2608.12700v1 Announce Type: new Abstract: Systems that generate GPU kernels with language models report high correctness rates. Those rates come from a single loose test: run the kernel on a few random inputs at one fixed shape and accept it if the output is close to a refe…
arXiv:2512.06547v4 Announce Type: replace-cross Abstract: Decoupled PPO has been a successful reinforcement learning (RL) algorithm to deal with the high data staleness under the asynchronous RL setting. Decoupled loss used in decoupled PPO improves coupled-loss style of algorith…
arXiv cs.LG
TIER_1English(EN)·Nikita Borodin, Maria Krylova, Artem Zabolotnyi, Dmitry Aspisov, Egor Shikov, Nikita Tyuplyaev, Oleg Travkin, Roman Alferov, Dmitry Vinichenko·
arXiv:2608.10850v1 Announce Type: new Abstract: We study continual pre-training (CPT) as a mechanism for adapting general-purpose large language models to specialized domains: mathematics, instruction, code, and natural text. Using singular value decomposition of weight matrices,…
arXiv:2608.10843v1 Announce Type: new Abstract: First-order concept synthesis asks a system to infer one formula that classifies labeled objects consistently across several finite relational structures. Every candidate can be evaluated exactly, but quantified first-order formulas…
arXiv:2608.08730v1 Announce Type: new Abstract: Large Language Models are deployed to multiple types of environments, from internet browsers to edge devices, and WebGPU serves as a modern cross-platform standard. The engines for browser-based LLM inference have proliferated, yet …
arXiv:2608.08878v1 Announce Type: cross Abstract: Transformer-based large language models (LLMs) achieve strong performance across many tasks, but their Key-Value (KV) cache grows linearly with sequence length, creating a severe memory bottleneck for long-context inference. Exist…
arXiv cs.AI
TIER_1English(EN)·Tejasvi C. Addagada·
arXiv:2608.09225v1 Announce Type: cross Abstract: The key-value (KV) cache is the primary throughput optimization in modern large language model (LLM) inference, enabling prefix reuse across requests. In multi-tenant deployments this cache is shared across tenants, creating a tim…
arXiv:2608.07974v1 Announce Type: new Abstract: Large language model (LLM) fine-tuning at the edge adapts the model to scenario-specific data while preserving privacy. Although existing studies proposed pipeline parallelism to address the limited memory and computing resources of…
arXiv:2502.11027v5 Announce Type: replace Abstract: Large language model (LLM) scaling inference is key to unlocking greater performance, and leveraging diversity has proven an effective way to enhance it. Motivated by the observed relationship between solution accuracy and meani…
arXiv cs.LG
TIER_1English(EN)·Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse·
arXiv:2608.06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests. LLM serving platforms today define response-la…
arXiv:2608.06763v1 Announce Type: new Abstract: Weight quantization for large-language-model inference must balance adaptive reconstruction levels with representations regular enough for efficient GPU execution. Uniform integers constrain each group to a linear grid. Low-bit floa…
arXiv:2608.06135v1 Announce Type: new Abstract: Large Language Models (LLMs) such as ChatGPT and Claude are widely used for information retrieval and problem-solving. Recent work has focused on improving scheduling algorithms to boost throughput while maintaining low latency. How…
arXiv cs.AI
TIER_1English(EN)·Guanqiao Qu, Shuo Chen, Qian Chen, Kin K. Leung, Xianhao Chen·
arXiv:2608.05926v1 Announce Type: cross Abstract: Edge inference is a promising paradigm to provide large language model (LLM) inference services in next-generation mobile networks. LLM inference mainly relies on two approaches: Autoregressive decoding (AD) generates output token…
arXiv cs.CL
TIER_1English(EN)·Sangwoo Ha, Hyunwoo Seo, Yurim Jo, Youngjin Moon, Hoi-Jun Yoo·
arXiv:2608.05303v1 Announce Type: cross Abstract: On-device deployment of Large Language Models (LLMs) has become essential for personalized edge applications. A primary bottleneck is external memory access (EMA) in feed-forward network (FFN) layers. Speculative decoding and mixt…
Weight quantization for large-language-model inference must balance adaptive reconstruction levels with representations regular enough for efficient GPU execution. Uniform integers constrain each group to a linear grid. Low-bit floating-point formats use a fixed exponent-mantissa…
Edge inference is a promising paradigm to provide large language model (LLM) inference services in next-generation mobile networks. LLM inference mainly relies on two approaches: Autoregressive decoding (AD) generates output tokens sequentially, resulting in long latency; Specula…
arXiv:2604.07396v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) inference on edge Neural Processing Units (NPUs) is fundamentally constrained by limited on-chip memory capacity. Although high-density embedded DRAM (eDRAM) is attractive for storing activation …
arXiv:2608.02947v1 Announce Type: cross Abstract: The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position. Aligned with this structure, we…
The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position. Aligned with this structure, we propose the per-RoPE-wavelength distance window: …
<h1><span>Abstract</span></h1><p><span>As the capabilities of frontier AI systems rapidly advance, there is growing interest in a multilateral agreement between nations to mitigate the risks posed by future, powerful systems. Any such agreement will require a quantifiable definit…
<p><span>Large language models often take actions running on one computer (via an agentic harness such as Claude Code or Codex), however the LLMs’ responses to prompts are computed on a different computer with GPU access. Could a malicious LLM gain control of the host machine whe…
X — Together (inference / OSS)
TIER_1English(EN)·togethercompute·
2/ New deep dive: Autoscaling endpoints for LLM inference.
Dedicated Inference can scale on eight metrics. inflight_requests is the default because it sees queue pressure before latency degrades.
We tested three autoscaling policies against the same load. Only one scaled.
The
AWS Machine Learning Blog
TIER_1English(EN)·Mona Mona·
Benchmark two 30B Mixture-of-Experts models, Qwen3-Coder-30B and NVIDIA Nemotron-3-Nano-30B, across G5, G6, G6e, and G7 GPU instances on Amazon SageMaker AI. Compare throughput, latency, and cost-per-token, and see how G7's NVIDIA Blackwell GPUs deliver measurable price-performan…
Latent Space (podcast video)
TIER_1English(EN)·Latent Space·
From pushing inference beyond 4,000 tokens per second to working with OpenAI on a new generation of ultra-fast AI infrastructure, Cerebras is betting that speed doesn’t just make models faster, it makes entirely new kinds of AI possible. In this episode, Cerebras co-founder and C…
Hacker News — AI stories ≥50 points
TIER_1English(EN)·saikatsg·
<p>In my previous article “<a href="https://medium.com/@andrewrites/get-llms-to-say-more-with-less-41636ce31268"><em>Get LLMs To Say More With Less</em></a><em>”</em>, we established the following goal: To train a small language model to write simpler English without sacrificing …
<h4>A prototype on Apple Silicon evolved into a production-grade, distributed, fully reproducible ML pipeline on AWS.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-_WmKj7PYS2penJBWgTe5g.png" /><figcaption>Source: Image by the author.</figcaption></figur…
Medium — Claude tag
TIER_1English(EN)·Sharvasuresh·
<h4>Practical techniques behind running large language models faster, cheaper, and on smaller hardware</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U1VMbNAt2L93jxDZe2qFeA.png" /></figure><p>2026 is again one of those years you realize that big tech is n…
<table> <tr><td> <a href="https://www.reddit.com/r/MachineLearning/comments/1wi68jg/gobench_evaluating_llms_on_the_game_of_go_r/"> <img alt="GoBench: Evaluating LLMs on the game of Go [R]" src="https://preview.redd.it/tbl17cn8ixph1.png?width=140&height=70&auto=webp&s=…
<blockquote> <p>⚡ <strong>TL;DR:</strong> Fine-tuning changes how a model behaves, not what it knows, so reach for it on style and format, not facts. Learn when prompting or RAG wins instead, and how to build a dataset that actually moves the needle.</p> </blockquote> <h2> Conten…
<h2> Architectural Decomposition of OpenArch: A PyTorch-Native Implementation Analysis </h2> <p>The landscape of Large Language Model (LLM) research is characterized by a rapid iteration cycle. While high-level libraries provide abstractions for inference and fine-tuning, the und…
<p>When a Python microservice misbehaves in production, you reach for logs and distributed traces. When an LLM-powered application misbehaves, most teams are flying blind. Tokens leak, latency spikes, costs explode — and all you have is a user complaint. That gap is what AI obser…
dev.to — LLM tag
TIER_1English(EN)·kapil Maheshwari·
<h2> Key takeaways </h2> <ul> <li>Streaming responses can reduce perceived latency but increase costs.</li> <li>Batching may save on processing but can introduce latency spikes.</li> <li>Understanding your user experience needs is critical for choosing between methods.</li> <li>E…
<h1> LLM Inference Optimization: Techniques for Faster and Cheaper AI </h1> <p><a class="article-body-image-wrapper" href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com…
<blockquote> <p><strong>TL;DR —</strong> Llama 4 Scout ships with a vendor-claimed 1.3 million token context window — big enough to hold a mid-size codebase or a full case file in one shot. Today's live probes found the model's structured-output extraction solid but wrapped in ma…
dev.to — LLM tag
TIER_1English(EN)·kapil Maheshwari·
<h2> Key takeaways </h2> <ul> <li>Implement circuit breakers to handle LLM outages gracefully.</li> <li>Fallback mechanisms can reduce user frustration during downtimes.</li> <li>Prioritize local inference models for critical tasks.</li> <li>Graceful degradation can enhance user …
<p>Okay, this is going to sound dumb, but the most expensive line item in one of my client pipelines last quarter was a model being asked to return JSON. Not to reason about anything. Just to take a support ticket, pull out six fields, and give them back as an object that <code>j…
<p>Teams are quietly rewriting their LLM integration code this quarter. Not for performance gains, not for new features — but because OpenAI's GPT-4o mini and Google's Gemini 1.5 Flash dropped pricing to levels that make older models look absurd. The threads on r/MachineLearning …
dev.to — LLM tag
TIER_1English(EN)·Nikhil raman K·
<h1> Query-Aware Code Compression for LLMs: 1.3M LOC to 1,500 Lines in 2.8s </h1> <h2> The Problem </h2> <p>You want to ask an LLM a question about your codebase. You have three options:</p> <ol> <li><p><strong>Paste everything</strong> — Repomix dumps your entire repo. For Next.…
dev.to — LLM tag
TIER_1English(EN)·LearnAI Resource·
<h1> The Hidden Costs of AI: Actually Benchmarking Your LLM Calls </h1> <p>You shipped an AI feature. It works. Users love it. Then your cloud bill shows up and suddenly you're rethinking life choices.</p> <h2> The Problem Nobody Talks About </h2> <p>Everyone benchmarks latency. …
<p>You ship an LLM feature. Weeks later you tweak a prompt, or the provider rolls the model forward under you, and something breaks — not loudly, not in a stack trace, just three answers that used to be right and now aren't. Nobody notices until a user does.</p> <p>I evaluate LLM…
<!-- SC_OFF --><div class="md"><p>One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots.</p> <p>A model is evaluated, a score is published, and we tend to talk about that score as if it describes a relatively stable object.…
<h2> 0. What I studied and what I found </h2> <p>I studied <strong>nano-vLLM as a small LLM serving engine</strong>: modeled its prefill and decode costs, traced the implementation, and compared the predictions with <strong>Qwen3-0.6B BF16 on one RTX 3090 (24 GB)</strong>. Its co…
<p>Imagine asking an AI coding assistant to help refactor a complex application. At first, it gives sharp, accurate responses. But 20 messages into the session, it suddenly forgets the architecture rules you set at the beginning, re-introduces previously fixed bugs, or hallucinat…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>Unlocking the inference performance of the Ascend 910B platform hinges on the complete adaptation of three layers—drivers, operators, and frameworks—each indispensable. This is a foundational insight that technical decision-makers must prioritize in domestic computing power re…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p><a href="https://docs.nvidia.com/dynamo/getting-started/quickstart" rel="noopener noreferrer">NVIDIA Dynamo</a> is an open-source, high-throughput, low-latency inference framework for deploying large-scale generative AI and reasoning models across multi-node, multi-GPU environ…
<h3> The Capacity Frontier </h3> <p>Self-hosting large language models (LLMs) is rarely a compute problem; it is a capacity problem. The weights must fit into fast, unified memory, or the system effectively grinds to a halt. The release of the <a href="https://www.apple.com/mac-s…
<!-- SC_OFF --><div class="md"><p>Hi, everyone,</p> <p>I’ve just released VoxGen, a lightweight native inference engine for VoxCPM2, written in Rust and using Vulkan compute instead of Python/PyTorch/CUDA.</p> <p><strong>Why VoxGen?</strong></p> <p>The main reason I started the p…
<blockquote> <p><strong>TL;DR —</strong> vLLM is the inference server most teams reach for when they need to serve open-weight LLMs at scale, thanks to PagedAttention and continuous batching. This episode breaks down what those two mechanisms actually do to GPU utilization, walks…
<p>The cheapest runtime is the one you never wake up at 3 AM to debug. After months of juggling free model endpoints, token meters, and a borrowed GPU, I stopped asking "which costs less" and started asking "which costs less when it fails."</p> <p>Free tiers are not a product dec…
<blockquote> <p><strong>TL;DR —</strong> llama.cpp is the C++ project that made running open-weight LLMs on ordinary hardware routine, via the GGUF format, block quantization, and CPU+GPU offload. This episode breaks down what those three mechanisms actually do and when reaching …
dev.to — LLM tag
TIER_1English(EN)·Prabhakar Chaudhary·
<h1> Prefix Sliding: Decoupling Memory from Reasoning Length in Test-Time Scaling </h1> <p>Test-time scaling has emerged as a primary frontier for increasing the capabilities of Large Language Models (LLMs). By allowing a model to generate longer reasoning traces—often referred t…
dev.to — LLM tag
TIER_1English(EN)·Athreya aka Maneshwar·
<p><em>Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. <a href="https://github.com/HexmosTech/git-lrc?utm_source=ratatop" rel="noopener noreferrer">Star git-lrc</a> to help devs discover th…
dev.to — LLM tag
TIER_1English(EN)·Prashant Lakhera·
<p>Most people use ChatGPT every day. But have you ever wondered what actually happens after you press Enter? The answer does not magically appear all at once.</p> <p>Behind the scenes, an LLM undergoes a complex inference process.</p> <p>And if you are learning LLM Engineering, …
<blockquote> <p><strong>TL;DR —</strong> Ollama turned running an open-weight model locally into a single command, and that simplicity is why it's now the default entry point for local AI. It's genuinely good for prototyping, private assistants, and air-gapped tools, but it isn't…
<p><em>Originally published at <a href="https://tekmag.thsite.top/omlx-the-open-source-menu-bar-llm-inference-server-trending-on-github/" rel="noopener noreferrer">https://tekmag.thsite.top/omlx-the-open-source-menu-bar-llm-inference-server-trending-on-github/</a></em></p> <p><st…
dev.to — LLM tag
TIER_1English(EN)·Vahid Aghajani·
<blockquote> <p>Originally published on <a href="https://software-engineer-blog.com/content/running-llms-locally-small-models-quantization-and-your-4gb-gpu?id=53" rel="noopener noreferrer">my blog</a>. Cross-posted here with a canonical link.</p> </blockquote> <p>I use <strong>Ge…
<p>The latest <code>llama.cpp</code> release, <code>b10481</code>, introduces substantial CUDA optimizations for dense models and refined handling of Mixture-of-Experts (MoE) architectures. This update aims to significantly boost inference performance by improving memory access t…
dev.to — LLM tag
TIER_1English(EN)·Mingxin Technology·
<p>When LLM inference encounters insufficient VRAM, optimization paths can be grouped into three categories: offloading the KV Cache to external storage, sharding and offloading model states, and restructuring attention computation from an I/O-aware perspective. These approaches …
<h2> The Memory Wall Problem </h2> <p>Running large language models efficiently requires more than just powerful GPUs. This post covers quantization (GPTQ, AWQ, GGUF), KV-cache optimization, speculative decoding, and how techniques like FlashAttention reduce memory bandwidth bott…
<p>Running LLMs inside macOS virtual machines on Apple Silicon has been painfully slow — until now. A team at Cua has discovered that the performance bottleneck isn't the hardware or the hypervisor, but a conservative GPU capability report that forces llama.cpp into slow code pat…
<h1> Model Quantization Explained: Shrinking LLMs Without Losing Their Mind </h1> <p>As we all know that when we try to load a 70B parameter model, how much time and space it takes just to load the model. To solve this issue, the concept of quantization was introduced.</p> <p>Thi…
Inception zaprezentowało Mercury 2.5 – model typu dyfuzyjnego LLM, który łączy okno kontekstowe 260 tys. tokenów z rekordową wydajnością na standardowym sprzęcie. # si # ai # sztucznainteligencja # wiadomości # informacje # technologia https:// aisight.pl/technologia/generat ywna…
