New LLM research tackles unlearning, acceleration, serving, and evaluation
ByPulseAugur Editorial·[100 sources]·
Multiple research papers explore advancements in large language model (LLM) capabilities and efficiency. "Nullify" introduces a training-free method for unlearning sensitive information from LLMs without compromising utility. DynaTE presents a hardware-software co-design to accelerate diffusion LLMs by dynamically adapting execution based on token states. Mooncake offers a disaggregated architecture for LLM serving, optimizing KVCache utilization to increase throughput and meet latency requirements. Other research focuses on reproducible LLM inference benchmarking, unbiased reward modeling from implicit feedback, and novel optimization techniques like CM-DPO for LLM planning and Clean for second-order training at linear memory cost.
AI
IMPACT
These papers introduce novel techniques for LLM unlearning, acceleration, serving efficiency, evaluation, and optimization, potentially improving model performance, privacy, and resource utilization.
RANK_REASON
Multiple research papers published on arXiv detailing new methods and architectures for LLMs.
arXiv:2610.23249v1 Announce Type: new Abstract: Small-to-large language model routing can reduce inference cost by escalating difficult inputs while retaining easier ones on a smaller model. Its success, however, depends on identifying examples that benefit from escalation, and t…
arXiv cs.AI
TIER_1English(EN)·Giovanni Vaccarino, Emanuele La Malfa, Dehai Min, Hanrong Zhang, Pier Luca Lanzi, Lu Cheng, Michael J. Wooldridge, Philip S. Yu, Sara Bernardini·
arXiv:2610.22568v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as formalizers: given a reasoning problem expressed in natural language, they produce a formal specification that a symbolic solver can then solve, with guarantees relative to that …
arXiv:2610.16352v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has emerged as a critical phase in the post-training pipeline of large language models (LLMs). As the frontier of LLM capabilities shifts from static reasoning to agentic tasks--characterized by…
arXiv:2610.23583v1 Announce Type: new Abstract: Long-term LLM assistants must repeatedly decide what prior context to expose and which model should answer as an interaction evolves. Existing memory systems and model routers usually make these decisions separately, although memory…
arXiv cs.LG
TIER_1English(EN)·Tingfeng Lan, Yusen Wu, Bin Ma, Zhaoyuan Su, Rui Yang, Tekin Bicer, Masahiro Tanaka, Olatunji Ruwase, Dong Li, Yue Cheng·
arXiv:2505.12242v4 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equa…
arXiv cs.LG
TIER_1English(EN)·Zhi Zhang, Arash A. Amini·
arXiv:2610.21363v1 Announce Type: cross Abstract: Test-time scaling can improve the performance of large language models by generating multiple answers to a question and aggregating them. Weighted ensembles extend this idea by drawing answers from several models, with the weights…
arXiv:2512.20159v2 Announce Type: replace-cross Abstract: Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to assess such code before deployment. As conventional rule-based metrics merely score c…
arXiv cs.AI
TIER_1English(EN)·Michael Hind, Basel Shbita, Bo Wu, Farhan Ahmed, Chad DeLuca, Nathan Fulton, David Cox, Dan Gutfreund·
arXiv:2603.22519v4 Announce Type: replace-cross Abstract: Textual Large Language Models (LLMs) provide a simple and familiar interface: a string of text is used for both input and output. However, the information conveyed to an LLM often has a richer structure and semantics, whic…
arXiv:2605.08581v2 Announce Type: replace Abstract: Online large language model (LLM) services process requests that share context but differ in prefix structure and arrival timing. Finite capacity couples request admission and key-value (KV) retention. This paper addresses one o…
arXiv:2608.01718v2 Announce Type: replace Abstract: Semantic caching, reusing responses to answer semantically similar queries, has seen growing adoption in LLM serving, offering faster responses and reduced costs. Yet existing schemes are fundamentally vulnerable to cache-collis…
arXiv:2610.16451v1 Announce Type: cross Abstract: Fine-tuning large language models at scale is increasingly limited by the memory cost of backpropagation. Zeroth-Order (ZO) optimizers reduce this burden by relying only on forward function evaluations, but they still exhibit a su…
arXiv:2610.18143v1 Announce Type: new Abstract: Prefix reuse improves LLM inference by sharing the KV cache of common prompt prefixes. This helps only if the scheduler can execute a matched branch in the current worker round. Schedulers assume that a matched prefix is ready, whic…
arXiv:2610.15228v1 Announce Type: new Abstract: Words are increasingly becoming an interface for action: large language models (LLMs) can now invoke tools, operate devices, and make decisions that affect the physical world. However, current benchmarks largely evaluate LLMs throug…
arXiv:2610.14745v1 Announce Type: new Abstract: Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quantization often suffers severe accuracy loss, while vector quantization (VQ) better…
arXiv:2610.14389v1 Announce Type: new Abstract: Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability. We directly measure this trade-off for 86 open-weight model checkpoints, evaluated on seven benchmarks and six hardware pla…
arXiv cs.LG
TIER_1English(EN)·Miko{\l}aj Pi\'orczy\'nski, Wojciech Masarczyk, Damian S\'ojka, Sebastian Cygert, Bart{\l}omiej Twardowski·
arXiv:2610.19892v1 Announce Type: new Abstract: Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shifts in CLIP-like contrastive models, its application to generative multimodal large …
arXiv cs.LG
TIER_1English(EN)·Casper L. Christensen, Kyle J. Ray, Dmitry Manning-Coe, Adam Shai, Paul Riechers·
arXiv:2610.19238v1 Announce Type: new Abstract: When writing code, the same computation can be expressed in different programming languages and through different implementation strategies. We show that the activations of LLMs capture this shared structure in reusable and composab…
arXiv:2610.20027v1 Announce Type: cross Abstract: Linters are widely used in code maintenance and real-time IDE editing to surface lint alerts related to performance, style, error-prone, and other patterns in source files. However, existing code benchmarks do not explicitly evalu…
arXiv:2609.32493v2 Announce Type: replace-cross Abstract: Distillation enables student language models to acquire new capabilities from expert teachers. However, integrating knowledge from multi-teacher, multi-domain demonstrations into a single student remains challenging. We st…
arXiv cs.AI
TIER_1English(EN)·Eunsu Kim, Haneul Yoo, Guijin Son, Hitesh Patel, Amit Agarwal, Alice Oh·
arXiv:2506.00482v2 Announce Type: replace-cross Abstract: LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can confl…
When writing code, the same computation can be expressed in different programming languages and through different implementation strategies. We show that the activations of LLMs capture this shared structure in reusable and composable abstract representations when they generate c…
Words are increasingly becoming an interface for action: large language models (LLMs) can now invoke tools, operate devices, and make decisions that affect the physical world. However, current benchmarks largely evaluate LLMs through static text responses, offline tool calls, or …
arXiv:2610.11284v1 Announce Type: cross Abstract: Diffusion-based LLMs (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) LLMs by enabling bidirectional parallel refinement, alleviating the sequential decoding bottleneck of AR generation. However, the…
arXiv:2407.00079v5 Announce Type: replace-cross Abstract: Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the und…
arXiv cs.LG
TIER_1English(EN)·Younwoo Choi, Leo Feng, Vincent Liu, Haanvid Lee·
arXiv:2610.10848v1 Announce Type: new Abstract: Accurate evaluation is central to selecting which LLM to deploy, yet testing a candidate on live traffic exposes real users to an unvetted model. Teams therefore evaluate candidates offline, on data produced by already-deployed mode…
arXiv:2610.10655v1 Announce Type: cross Abstract: Large Language Models (LLMs) inevitably internalize substantial amounts of sensitive or private information during pre-training, while LLM unlearning aims to selectively erase specific knowledge to prevent privacy leakage with min…
arXiv cs.LG
TIER_1English(EN)·Rabimba Karanjai, Qun Gu, Hemanth Hegadehalli Madhavarao, Wenhuan Sun, Xiaojiao Yu, Suryabhan Singh Hada, Libin N. George, Uma Kona, Richard Williamson, Linsey Pang, Prakhar Mehrotra·
arXiv:2610.09219v1 Announce Type: cross Abstract: Direct Preference Optimization (DPO) treats all constraint violations equally: a $1 budget overshoot and a $1,000 overshoot induce the same training signal. It is also susceptible to length and style bias when preference pairs com…
arXiv cs.LG
TIER_1English(EN)·Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek, Guang Lu, Jo\~ao Carvalho·
arXiv:2610.09778v1 Announce Type: cross Abstract: Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system state. We present the Sequential Isolation Methodology, a controlled benchmarking an…
arXiv:2610.04204v2 Announce Type: replace-cross Abstract: Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature, whereas full-curvature methods such as SOAP can accelerate convergence at p…
arXiv:2610.07362v1 Announce Type: new Abstract: We evaluate large language models (LLMs) in multi-turn interactions through their time-to-event: the number of interaction steps required to produce an event of interest, such as a successful jailbreak or agentic task completion. Un…
arXiv cs.AI
TIER_1English(EN)·Jinhyeok Kim, Hye-Young Jung·
arXiv:2610.08153v1 Announce Type: cross Abstract: Multiple-choice question answering (MCQA) is commonly used to evaluate large language models under the assumption that one of the provided options is correct, typically using answer-selection accuracy. However, in real deployments…
arXiv:2603.23184v2 Announce Type: replace-cross Abstract: Despite the success of reinforcement learning from human feedback (RLHF), existing reward modeling methods largely rely on explicit feedback, which is costly to collect and difficult to scale. This work studies implicit re…
arXiv cs.AI
TIER_1English(EN)·Abhinav Sudhakar Dubey (University of California Santa Cruz), Scott Sirri (University of California Santa Cruz), Vaggos Chatziafratis (University of California Santa Cruz), C. Seshadhri (University of California Santa Cruz)·
arXiv:2610.07125v1 Announce Type: cross Abstract: While open-weight models have enjoyed steady progress in capabilities and wide adoption across multiple domains, their safety remains an important concern. One key feature is the ability to refuse or deflect harmful, malicious, or…
arXiv:2610.06993v1 Announce Type: cross Abstract: Evolution Strategies (ES) enable memory efficient full parameter fine-tuning of large language models (LLMs) using only forward computation. However, standard ES uniformly averages rewards across problems and compresses problem le…
Given a task and an evaluator, a language model can rewrite a candidate program while a search loop decides which rewrites survive, offering a practical route to algorithm discovery. But that loop is governed by five constants set by hand: which parent to select, how hard to muta…
Modern LLM training increasingly relies on heterogeneous data sources spanning different domains, tasks, preference distributions, and difficulty levels. We study dynamic minimax regret for group-distributionally robust LLM post-training under instantaneous mini-batch-only bandit…
arXiv:2610.03430v1 Announce Type: new Abstract: Efficient request scheduling is increasingly important for reducing completion time in large language model (LLM) serving. Size-based policies such as Shortest Job First prioritize shorter requests, but output lengths are unknown be…
arXiv cs.LG
TIER_1English(EN)·Arda Fazla, Antesh Upadhyay, Ege C. Kaya, M. Berk Sahin, Abolfazl Hashemi·
arXiv:2610.02355v1 Announce Type: new Abstract: Increasing the batch size during training is a common practice in large language model (LLM) pretraining, yet the theoretical justification behind its success is not well understood. Analyses of stochastic optimization often assume …
arXiv:2610.03316v1 Announce Type: new Abstract: Designing effective heuristics for diverse combinatorial optimization problems requires substantial expertise and repeated search. Large language models (LLMs) automate heuristic generation and refinement, but heuristic search typic…
arXiv:2610.02910v1 Announce Type: new Abstract: Suppressing a small set of routed experts can weaken the safety behavior of a sparse Mixture-of-Experts (MoE) language model without retraining. Which experts to suppress is therefore a security question, and the usual answer is act…
arXiv:2610.02462v1 Announce Type: cross Abstract: LLM compression reduces inference costs and memory requirements, but selecting a method and configuration remains largely empirical because comparable resource reductions can produce different capability losses. We systematically …
arXiv cs.LG
TIER_1English(EN)·Hongyu Chen, Xinyi Luo, Ming Zhao, Lin Tang, Zihan Xu, Jing Li, Yuxuan Wang, Haoran Deng, Wei Zhang·
arXiv:2610.00436v1 Announce Type: new Abstract: Online batch selection fine-tunes a language model on the most useful part of each candidate batch. Selectors that match the gradient of the candidate batch are attractive because they need no held-out data, yet they rarely beat tra…
arXiv:2610.01967v1 Announce Type: new Abstract: As large language models (LLMs) keep growing in size and complexity, their training frameworks evolve at a rapid pace as well. Therefore, continuous integration (CI) is critical for maintaining the quality and stability of these fra…
arXiv:2610.02067v1 Announce Type: new Abstract: While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify \textbf{adaptation imbalance}: a few singular directions dominate the traine…
arXiv cs.LG
TIER_1English(EN)·Cristian McGee, El Houcine Bergou, Aritra Dutta·
arXiv:2610.02190v1 Announce Type: new Abstract: Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder o…
arXiv cs.LG
TIER_1English(EN)·Jichao Jiang (University of Central Florida), Cristian McGee (University of Central Florida), El Houcine Bergou (Mohammed VI Polytechnic University), Hanqin Cai (University of Central Florida), Aritra Dutta (University of Central Florida)·
arXiv:2610.02199v1 Announce Type: new Abstract: Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-orde…
arXiv cs.LG
TIER_1English(EN)·Geon-Woo Kim, Joon Ha Kim, Daehyeok Kim·
arXiv:2610.00687v1 Announce Type: cross Abstract: Hardware-operable failures (HOFs) interrupt large language model (LLM) training but permit recovery on the same hardware without reset, repair, or replacement. Existing recovery systems nevertheless reload checkpoints, recompute l…
arXiv:2610.01537v1 Announce Type: new Abstract: Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fun…
Large Language Models (LLMs) are increasingly used for automated algorithm design. However the computational cost of evaluating the generated algorithms can be excessive. We consider the common LLM-driven automated algorithm design (LLM4AD) setting in which a candidate algorithm …
arXiv:2609.39866v1 Announce Type: new Abstract: Open-weight LLMs are released not only as fixed products but also as substrates for downstream fine-tuning. This openness, however, creates legal and ethical risks because users may misuse fine-tuning to instill illicit knowledge or…
arXiv:2609.40148v1 Announce Type: new Abstract: Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss. We study this dependence in noisy online SGD with linear random f…
arXiv:2610.01896v1 Announce Type: cross Abstract: Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies. Theoretical understanding of how this staleness affects convergenc…
arXiv cs.AI
TIER_1English(EN)·Haozhan Tang, Xiuqi Zhu, Xinyin Zhang, Boxun Li, Virginia Smith, Kevin Kuo·
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater representational plasticity, Low-Rank Adaptation (LoRA) can match or surpass FFT perform…
arXiv:2609.38767v1 Announce Type: cross Abstract: Training data attribution (TDA) estimates the contribution of individual training examples to model outputs. Most scalable TDA methods rely on per-example gradients, whose computation and use at LLM scale pose challenges in effici…
arXiv cs.AI
TIER_1English(EN)·Huaiyu Fu, Heng Cao, Hao Wang, Jian Ya, Tao Chen·
arXiv:2609.38805v1 Announce Type: cross Abstract: LLM agents often admit multiple high-quality solutions to the same task, differing in reasoning structure, tool-use pattern, or interaction trajectory. Yet existing notions of diversity in LLM post-training are mostly implicit, ar…
arXiv:2609.40286v1 Announce Type: cross Abstract: Unlearning a fact in one language does not guarantee its removal in others as changing the query or even the requested answer language can reopen seemingly forgotten knowledge -- a cross-lingual loophole. The most straightforward …
arXiv:2511.10333v2 Announce Type: replace-cross Abstract: Training large language models (LLMs) at scale incurs substantial communication overhead, while static gradient compression cannot adapt to gradient evolution and may degrade model quality. We propose EDGC, an entropy-driv…
arXiv cs.AI
TIER_1English(EN)·Chenhao Ye, Huaizheng Zhang, Mingcong Han, Baoquan Zhong, Xiang Li, Qixiang Chen, Xinyi Zhang, Weidong Zhang, Kaihua Jiang, Wang Zhang, He Sun, Wencong Xiao, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau·
arXiv:2604.09107v2 Announce Type: replace-cross Abstract: Modern LLM reinforcement learning (RL) workloads require a high-performance weight transfer system to scale training across heterogeneous compute resources. However, efficiently transferring terabyte-scale model weights ac…
arXiv cs.CL
TIER_1English(EN)·Srishti Ginjala, Eric Fosler-Lussier, Srinivasan Parthasarathy·
arXiv:2609.38976v1 Announce Type: new Abstract: Demographic fairness gaps in automatic speech recognition are almost always reported from a single training run. We fine-tune the Q-former projector and LoRA adapters of a speech LLM at five audio compression factors and six random …
Demographic fairness gaps in automatic speech recognition are almost always reported from a single training run. We fine-tune the Q-former projector and LoRA adapters of a speech LLM at five audio compression factors and six random seeds, holding the encoder, base decoder, data a…
arXiv:2609.36659v1 Announce Type: new Abstract: The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these studies treat the observed behaviors only as byproducts in on-policy training, over…
arXiv:2609.36830v1 Announce Type: new Abstract: Fully asynchronous reinforcement learning (RL) improves resource utilization in large language model post-training by overlapping rollout generation with policy optimization, but it also introduces policy lag as trajectories are gen…
arXiv cs.AI
TIER_1English(EN)·Pengyu He, Yan Zhang, Ruien Li, Guangwen Yang·
arXiv:2609.36662v1 Announce Type: cross Abstract: The pre-training of Large Language Models (LLMs) is increasingly conducted across multiple data centers. As training scales to a larger number of accelerators, the fraction of time spent on computation decreases, while the fractio…
arXiv:2609.37169v1 Announce Type: cross Abstract: Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since additional serial compute yields little further downstream improvement and can even …
arXiv:2609.37858v1 Announce Type: cross Abstract: Many localized large language model (LLM) unlearning methods select a small parameter subset from a localization signal and keep it fixed during optimization. The parameters most associated with a target, however, need not be the …
arXiv cs.LG
TIER_1English(EN)·Haozhan Tang, Hao Kang, Han Cai, Song Han, Chenyan Xiong·
arXiv:2609.37852v1 Announce Type: new Abstract: Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta…
The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these studies treat the observed behaviors only as byproducts in on-policy training, overlooking their potential to serve as optimization…
Reinforcement learning (RL) post-training for large language models (LLMs) coordinates multiple models across generation, inference, and training on GPU clusters. Several factors may change during a run, including resource availability, sequence length, memory pressure, and stage…
arXiv stat.ML
TIER_1English(EN)·Aditya Kulkarni, Davide Sartor, Carlos Soto, Nathan Wycoff·
arXiv:2610.18504v1 Announce Type: new Abstract: Recently, Morris et al. introduced TinyLoRA, a strategy for LLM fine-tuning which introduces arbitrarily few parameters, and demonstrated accuracy improvements using less than $10$ parameters rather than the ${\gtrsim 10,000}$ typic…
arXiv:2610.00431v1 Announce Type: new Abstract: Continual parameter-efficient fine-tuning for large language models (LLMs) must balance retention of previously acquired knowledge, adaptation to new tasks, and strict parameter budgets. We present \textbf{ChainLoRA}, a replay-free …
<p>When we talk about giving an AI agent 'access' to a system, our mental model usually defaults to software. We think of an agent reading a Jira ticket, querying a Postgres database, or sending a message via Slack. But the boundary between digital intent and physical reality is …
<p>Recently, I have been building a local data analysis stack pairing <strong>LM Studio (<code>Qwen3-8B</code>)</strong> with custom <strong>Model Context Protocol (MCP)</strong> servers to query and cross-analyze over 180 normalized Japanese government and corporate datasets.</p…
Medium — fine-tuning tag
TIER_1Deutsch(DE)·William Antônio Siqueira·
<h2> Understanding Free Tiers </h2> <p>Free tiers of Large Language Model (LLM) APIs offer a starting point for experimenting with AI capabilities without significant upfront costs. However, these tiers often come with limitations such as request quotas, response size, and model …
dev.to — LLM tag
TIER_1English(EN)·Dhanush Kumar Sivaji·
<p>I've spent 8+ years building React/TypeScript front ends. Now I'm learning applied AI in public, working toward AI Engineer, then AI Solution Architect, and (I hope) AI Forward Deployed Engineer. Before tokens, RAG or agents, I wanted one map: what actually happens when a user…
<p><em>Part 7 of a series. Previously: <a href="https://dev.to/USERNAME/PART-6-SLUG">Part 6 — The Orchestration Ceiling</a>.</em></p> <p>Part 6 made the team's execution reliable: by moving the control loop out of the model and into deterministic code, the orchestrator could carr…
dev.to — LLM tag
TIER_1English(EN)·Swathi Rajagopal·
<p><a href="https://www.securekloud.com/generative-ai-services" rel="noopener noreferrer">Large Language Models (LLMs)</a> have made it easier to extract structured information from unstructured documents. A developer can send an invoice, bank statement, or application form to a …
dev.to — LLM tag
TIER_1English(EN)·Malek AIT OUARET·
<p>Pseudocode is how we teach algorithms, yet it has never been executable. You can write "repeat divisor from 2 to 50"<br /> on a whiteboard, but not run it. I wanted to change that without inventing yet another syntax, so I built<br /> <strong>Pseudo.NET</strong>: you write pse…
dev.to — LLM tag
TIER_1English(EN)·Vainamoinen | Pulsed Media·
<p><em>9.99 € a month buys 9,990,000 credits on one API key: about 6 million input tokens on GLM-5.3, or about 37 million on the fast default model, from curl, your own code, Claude Code or Gemini CLI. Under the current rules the credits don't expire.</em></p> <p><em>I'm Väinämöi…
<p><em>Part 6 of a series. Previously: <a href="https://dev.to/USERNAME/PART-5-SLUG">Part 5 — Distributing the Workers and the Memory Budget</a>.</em></p> <p>Part 5 ended with the 77 GB coordinator finally running as it should — resident, no thrashing, the memory problem resolved…
<p>Part 4 ended against a wall. A coordinator capable enough to be worth running — the 77 GB <code>gpt-oss:120b</code> — could not share a machine with the workers it had to call; every delegation forced roughly 100 GB of model-swapping, and a greeting took nearly five minutes. T…
<p>Tuần này trên Hacker News có một project khá thú vị tên Janas: chạy LLM lớn hơn RAM bằng cách stream các expert từ SSD. Đọc xong mình nhớ lại hồi đầu năm, khi cố nhét một model MoE hơn 100B params vào con laptop 32GB RAM. Lần đầu thì máy treo cứng, swap kín ổ. Lần sau thì chạy…
<h1> Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison </h1> <p>Run any model on your hardware — no API keys, no cloud, complete privacy.</p> <h2> Introduction </h2> <p>Local LLMs have matured. You can now run 7B-70B models on consumer hardware with near-cloud perfor…
<h1> Fine-Tuning LLMs on Consumer Hardware: LoRA Guide 2024 </h1> <p>Train custom models on your GPU using LoRA/QLoRA — no cloud credits needed.</p> <h2> Introduction </h2> <p>Fine-tuning used to require A100s and thousands of dollars. With LoRA (Low-Rank Adaptation) and QLoRA (Q…
<h2> TL;DR </h2> <p>Running a language model on a device, a tablet, a handset, an edge box, is not just the cloud without the network. It is a different budget. Memory is fixed, precision is a lever, latency includes a cold start, and there is no autoscaler to hide your mistakes.…
<h1> Stop Wasting Hours on Prompt Engineering: Your Self-Hosted LLM Just Got Smarter </h1> <p>Running your own LLM locally is freedom. Running Llama-3 or Mistral on your infrastructure means no API costs, no rate limits, no vendor lock-in. But there's a hidden cost nobody talks a…
dev.to — LLM tag
TIER_1English(EN)·Muthali Ganesh·
<p>Claude Opus 5.5 fixed all 18 critical defects in GoML's LatticeBench evaluation and resolved six of seven <code>very_hard</code> issues. The evaluation also revealed a surprising weakness: despite its performance on complex security defects, the model struggled with straightfo…
dev.to — LLM tag
TIER_1English(EN)·LLM API Cost Calculator Token·
<p>Most LLM cost comparisons collapse a model to one number: <strong>$X per million<br /> input tokens</strong>. That number is the price of a cache <em>miss</em>. Once prompt caching<br /> is on, most of the tokens you send on every request are billed at the cached-read<br /> ra…
<p>You can prototype on LLMs without paying anything - but "free tier" means very different things depending on the provider. Some give permanent rate-limited access, some give a monthly credit, and some only free a handful of models.</p> <p>Here are nine LLM APIs that need <stro…
<p>Alright folks, let's talk about Large Language Models. Specifically, my recent adventure trying to coerce one into giving me genuinely <em>actionable</em> advice for a gnarly, old Go microservice. We've all seen the headlines, right? "AI optimizes your code!" "Let an LLM find …
dev.to — LLM tag
TIER_1English(EN)·Abhash Chakraborty·
<p>I needed a model for one narrow decision. When a new fact about someone comes in, does it replace something already known? "I moved to Berlin" should retire "I live in Pune". "Watched two episodes of The Bear" should leave "works at Infosys" alone.</p> <p>It sounds trivial, bu…
<table> <tr><td> <a href="https://www.reddit.com/r/OpenAI/comments/1x0q5u5/a_reproducible_costpertask_method_for_llm_pricing/"> <img alt="A reproducible cost-per-task method for LLM pricing: sticker price per token ranked models differently in my own two benchmarks too" src="http…