RadixAttention
PulseAugur coverage of RadixAttention — every cluster mentioning RadixAttention across labs, papers, and developer communities, ranked by signal.
2 day(s) with sentiment data
-
SGLang inference engine boosts LLM performance with token-level KV cache
SGLang is a new open-weight AI inference engine designed to significantly improve performance for specific LLM workloads. It utilizes a novel RadixAttention mechanism that caches KV cache at the token level, enabling hi…
-
Mingxin FX100 boosts LLM inference with KV Cache reuse · 2 sources tracked
Mingxin FX100 has demonstrated significant performance improvements in multi-turn dialogue scenarios for large language models. By implementing KV Cache reuse strategies, which involve caching key-value tensors from pre…
-
UnfoldML integrates RadixAttention to boost LLM efficiency
UnfoldML has introduced RadixAttention, a new method for improving the efficiency of large language models. This technique is designed to reduce the computational cost associated with attention mechanisms, which are a c…
-
UnfoldML optimizes LLM inference with RadixAttention KV caching
UnfoldML has introduced RadixAttention, a new KV caching strategy designed to optimize the prefill phase of LLM inference. This method utilizes a radix tree data structure to efficiently store and share common prefixes …
-
New techniques boost small LLM Bash generation and speed up AI inference
Researchers have developed a technique called grammar-constrained decoding to improve the Bash command generation capabilities of small language models. This method enhances accuracy and safety, transforming natural lan…