LRU
PulseAugur coverage of LRU — every cluster mentioning LRU across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
WAKMan optimizes LLM KV cache management for diverse workloads
Researchers have developed WAKMan, a novel workload-aware KV cache manager designed to optimize Large Language Model (LLM) server performance. Traditional eviction policies like LRU and S3FIFO struggle with diverse prod…
-
New research tackles KV cache optimization for LLM efficiency
Multiple research papers explore novel techniques for optimizing the KV cache in large language models to improve efficiency and reduce latency. FlashOffloading proposes a host-DRAM-centric architecture to fully offload…
-
SpecMem system accelerates AI agent tool calls with live memory management
Researchers have developed SpecMem, a novel system designed to accelerate agentic tool calling by implementing live memory management. This system addresses the latency issues in tool-calling agents by using speculative…
-
llama.cpp v0.7.0 enhances GPU caching for faster MoE expert inference
The latest release of llama.cpp, version 0.7.0, introduces a new GPU cache for Mixture-of-Experts (MoE) models. This feature, accessible via the --moe-cache-mib option, allows experts to run on the GPU from an LRU cache…
-
New NC-SSM models overcome limitations of Mamba, S4D, and LRU
Researchers have introduced Non-Commutative State Space Models (NC-SSM) to address limitations in existing selective state space models like Mamba. These new models, including SO(3)-SSM and Cayley-SSM, extend state tran…
-
New research improves State Space Models with depth recurrence and input reshaping
A new research paper introduces two methods to enhance State Space Models (SSMs), making them more efficient and performant. The first method, depth recurrence, reduces the memory footprint of SSMs without sacrificing p…
-
New research explores memory's impact on routing games and introduces 'Recall Braess Paradox'
A new research paper explores routing games where travelers' route choices are influenced by memory and recall mechanisms, rather than solely fixed external factors. The study introduces the concept of a Forgetful Wardr…
-
AI agents to report tool progress for better performance
Researchers have developed a new method for AI agents to better manage their progress when using external tools. Current systems often guess how long a tool will take, leading to inefficient use of GPU memory. The propo…
-
LRU Caching Proves Surprisingly Resilient Against Advanced AI Cache Methods
A recent article suggests that Least Recently Used (LRU) caching strategies are more effective than previously understood, even when compared to advanced KV-cache techniques used in AI models. The author highlights that…
-
New H-MC caching policy offers improved regret guarantees over LRU/LFU
Researchers have developed a new caching policy called H-MC that aims to improve upon existing methods like LRU and LFU. Unlike previous policies such as LeCar and Cacheus, which can suffer from linear regret against ce…
-
Developers can fix LLM API 429 errors with caching
Developers encountering frequent HTTP 429 "Too Many Requests" errors from free-tier LLM APIs can often resolve the issue by implementing a cache. These errors, especially when occurring during peak hours, may stem from …
-
LLM Cache Study: LFU Outperforms Other Eviction Policies, But Effectiveness Limited
A new study published on arXiv evaluates various eviction policies for Large Language Model (LLM) caches, finding that the Least Frequently Used (LFU) policy performs best among those tested. The research, conducted usi…
-
MoE expert caching evaluation methods found to be misleading
Researchers have identified critical flaws in trace-driven evaluation methods for Mixture-of-Experts (MoE) models, which can lead to misleading conclusions about expert caching policies. The study highlights how replay …
-
New analysis explores caching strategies for Genetic Programming Symbolic Regression
This paper analyzes caching strategies for Genetic Programming Symbolic Regression (GPSR) to reduce computational runtime. The study explores memory-runtime trade-offs of various caching mechanisms, including LRU and FI…
-
LLM KV Cache Management Needs Rethink Beyond Simple Eviction
The KV cache in LLM inference servers is often managed with a simple LRU or capacity-based eviction policy, which is an inadequate approach for modern workloads. This strategy fails to account for the varying future val…
-
Phionyx introduces deterministic AI runtime for enhanced governance
Researchers have introduced Phionyx, a novel AI runtime architecture designed for deterministic behavior and enhanced governance. This architecture treats LLM outputs as noisy measurements, employing deterministic state…
-
KV cache eviction protection proves more vital than scoring
Researchers have developed a new method for managing KV cache eviction in large language models, finding that structural protection is more critical than scoring algorithms. Their study on transformer models revealed th…
-
Looped SSMs improve time series classification with depth-recurrence
Researchers have introduced Looped SSMs, a novel approach to State Space Models for time series classification. This method enhances performance by applying depth-recurrence, where model blocks are reused across layers,…
-
Apple researchers unveil SpecMD for faster MoE model inference
Apple's machine learning research team has published a paper detailing SpecMD, a new framework for evaluating Mixture-of-Experts (MoE) model caching policies. Their experiments show that traditional caching assumptions …
-
New ML-based GPU caching algorithm LCR boosts LLM inference speed
Researchers have developed a new GPU caching algorithm called Learning-Augmented LRU (LALRU) designed to improve efficiency during AI inference. This algorithm integrates learned predictions with caching policies to ens…