Flashattention
PulseAugur coverage of Flashattention — every cluster mentioning Flashattention across labs, papers, and developer communities, ranked by signal.
15 day(s) with sentiment data
-
llama.cpp adds OCP FP8 support for NVIDIA GPUs, boosting performance
The llama.cpp project has introduced support for OCP FP8 (Octet-Compressed Precision Floating Point 8-bit) on CUDA, enhancing performance for NVIDIA GPUs. This update includes optimizations for General Matrix Vector Mul…
-
New attention method scales 3D shape correspondence to high-resolution surfaces
Researchers have developed a new method called Geo-FlashAttention to improve the efficiency of shape correspondence algorithms for high-resolution 3D surfaces. By reformulating spatial-spectral coupling as an attention …
-
LLMs' memory evolution: From Attention to 3-bit caches
This article delves into the technical advancements that have enabled Large Language Models (LLMs) to handle longer contexts and improve efficiency. It traces the evolution from the foundational 'Attention Is All You Ne…
-
Together Compiler automates AI model optimization for performance gains
Together has launched Together Compiler, a new tool designed to automatically optimize inference kernels for AI models. This system utilizes agents that adapt to specific models, hardware, and traffic patterns. Early re…
-
ZYFA boosts AMD MI355X FP8 FlashAttention speed by up to 1.99x
Researchers have developed ZYFA, a new set of prefill attention kernels for AMD Instinct MI355X GPUs that leverage FP8 precision for improved speed and accuracy. ZYFA employs novel techniques, including a simplified pro…
-
Step-dLLM boosts diffusion LLM inference speed with adaptive sparse attention
Researchers have developed Step-dLLM, a novel method to improve the efficiency of diffusion large language models (dLLMs) during inference. Unlike autoregressive models, dLLMs cannot use KV caching and recompute attenti…
-
Together AI unveils Polymorphic Inference for adaptable AI model deployment
Together AI has introduced a new research agenda called Polymorphic Inference, designed to address the rapid evolution of AI models, architectures, and hardware. This approach contrasts with traditional Static Inference…
-
Formalizing GPU Kernels in Lean for Deep Learning Correctness
Researchers have developed a method to formally verify the correctness of GPU kernels used in deep learning by modeling Parallel Thread Execution (PTX) in the Lean proof assistant. This approach enables mathematical rea…
-
NVIDIA H200 GPU offers cost advantages over H100 for memory-bound AI tasks
The NVIDIA H200 GPU offers improved memory bandwidth compared to the H100, making it more efficient for memory-bound tasks like LLM inference. While the H200's compute power is similar to the H100, its higher memory thr…
-
KernelSeek AI agent optimizes CUDA kernels with questions, beats baselines
Researchers have developed KernelSeek, a novel question-driven AI agent designed to optimize CUDA kernels for GPU performance. Unlike existing agents that rely on extensive toolkits or fixed policies, KernelSeek uses ta…
-
FlashAttention kernel deduced from SQL using new extensions
A demonstration showcases the automatic deduction of a FlashAttention kernel using approximately 25 lines of SQL. This process leverages three portable SQL extensions being developed, specifically highlighting the xarra…
-
New Merge and Branch method boosts LLM fine-tuning efficiency for recommendations
Researchers have developed a new method called Merge and Branch (M&B) to make supervised fine-tuning of large language models (LLMs) more efficient for recommendation systems. M&B compresses historical item information …
-
New algorithm accelerates long-context transformers with relative-position attention
Researchers have developed a new algorithm that significantly speeds up and reduces memory usage for transformers handling long contexts, particularly those using relative-position attention mechanisms. This method inte…
-
LLM Serving Economics Revolutionized by KV Cache Innovations and Rust Rewrites · 1 source tracked
Recent research and engineering advancements are significantly improving LLM serving economics. Innovations include optimizing KV cache management through compaction and quantization, with new methods like role-aware qu…
-
INTRA sparse attention boosts transformer efficiency with novel relay-communication
Researchers have introduced INTRA, a novel sparse attention mechanism designed to improve the efficiency of transformer models. INTRA employs an interleaved, non-contiguous token approach, viewing attention as a relay-c…
-
New method folds attention biases into keys for efficient KV cache operation
Researchers have developed a method to fold attention biases into keys, enabling compacted KV caches to run on standard kernels. This technique, which involves refitting values and adding a bias to kept keys, results in…
-
New LLM Fine-Tuning Method Boosts Recommendation System Efficiency
Researchers have developed a new method called Merge and Branch (M&B) to make supervised fine-tuning of large language models (LLMs) for recommendation systems more efficient. M&B compresses item identifiers and text fr…
-
Triage method optimizes attention tiles in video diffusion transformers
Researchers have developed a new method called Triage for optimizing attention mechanisms in video diffusion transformers (DiTs). This training-free approach categorizes attention tiles into three tiers: skipped, skimme…
-
Engineers' guide to LLMs covers Transformers, RAG, and evaluation pitfalls
This paper provides a plain-language overview of large language models (LLMs) for engineers, detailing their underlying architecture and training processes. It explains key concepts such as the Transformer architecture,…
-
NVIDIA RTX 3000 GPUs get performance boost with firmware exploit
A tool called gpu-unlock has been developed to increase the performance of certain NVIDIA RTX 3000-series GPUs, specifically those based on the GA102, GA103, GA104, and GA106 chips. This tool exploits a buffer overflow …