half-precision floating-point format
PulseAugur coverage of half-precision floating-point format — every cluster mentioning half-precision floating-point format across labs, papers, and developer communities, ranked by signal.
- instance of single-precision floating-point format 90%
- used by graphics processing unit 70%
- instance of Int8 70%
- used by Int8 70%
- used by single-precision floating-point format 70%
- instance of bfloat16 70%
- instance of Fp8 70%
- used by Flashattention 70%
- instance of INTS8 70%
- used by central processing unit 60%
- competes with INTS8 60%
- instance of Int4 60%
7 day(s) with sentiment data
-
New neural compression framework enables real-time processing of fiber-optic sensing data
Researchers have developed STeP-C, a neural compression framework designed for real-time processing of large-scale distributed fiber-optic sensing data. This system addresses the challenges of high-volume data generatio…
-
New method improves BF16 convergence speed in RL post-training
A new research paper proposes a method to improve the convergence speed of reinforcement learning (RL) post-training using bfloat16 (BF16) precision. The study identifies "weight sticking" as a cause for slower converge…
-
Edge AI framework optimizes vision transformers for connected vehicles
Researchers have developed a new framework for optimizing the deployment of vision transformer models in connected and automated vehicles (CAVs). This framework addresses the limitations of static split-inference method…
-
LLM Inference: KV Cache Fills Before Compute Limit on Consumer GPUs
The article explains the concept of the "ridge" in LLM inference, which refers to the point where a model's computation becomes limited by memory bandwidth rather than processing power. It details how this "ridge" is af…
-
Local LLM Memory Needs: Weights, KV Cache, and Usable RAM Explained
Estimating the memory required for local LLMs involves calculating the sum of model weights, the KV cache, and overhead, then comparing this to the usable memory available on a device. Quantization significantly impacts…
-
Self-supervised SAR-optical encoder performance analyzed on x86 hardware
Researchers have evaluated the performance of a self-supervised vision transformer encoder for Earth observation tasks when run on commodity x86 hardware. The study found that INT8 quantization had a negligible impact o…
-
New method enables on-device LLM fine-tuning using phone NPUs
Researchers have developed a novel method for on-device fine-tuning of large language models (LLMs) using a phone's neural processing unit (NPU). This system, called First-Order LoRA Training, enables LLMs to learn from…
-
LLM KV Cache VRAM Calculator: Fitting Large Contexts into Limited Memory
A technical blog post details how to calculate the Video RAM (VRAM) required for large language model context lengths, focusing on the KV cache. It provides a formula to estimate VRAM needs based on model architecture a…
-
PrismQuant framework optimizes LLM quantization for improved performance
Researchers have introduced PrismQuant, a novel framework designed to optimize grouped quantizers for large language models. This method focuses on aligning activation eigenspaces with the quantizer's subspace, leading …
-
Bonsai quantization shrinks LLMs to 5.9GB with 98.2% intelligence retained
A new method for quantizing large language models, dubbed "Bonsai," has been developed, significantly reducing model size while retaining a high percentage of intelligence. This technique compresses 27B-class models to …
-
LLM KV cache, not weights, drives VRAM needs for long contexts
The memory requirements for running large language models (LLMs) extend beyond just model weights, with the KV cache being a significant factor, especially for long context windows. For instance, a 7B parameter model li…
-
llama.cpp b11544 release optimizes float conversion, updates types
The llama.cpp project has released version b11544, which includes optimizations for converting single-precision floating-point numbers to half-precision floating-point format on the ggml-cpu architecture. This update al…
-
Whisper dictation accuracy improved with prompt engineering and post-processing
The author details their experience improving the accuracy of the Whisper speech-to-text model for personal dictation. They found that many perceived errors were not mishearings but rather issues with filler words, spok…
-
AI's convenience in tasks from drafting emails to model fine-tuning discussed
The convenience of AI is highlighted, with examples ranging from drafting emails to debugging code and outlining marketing strategies. The discussion also touches upon the memory requirements for fine-tuning a 7B model,…
-
New LLM Serving System Optimizes KV-Cache Memory with Dual Precision
Researchers have developed DPS, a dual-precision LLM serving system that dynamically adjusts model precision to optimize KV-cache memory. By switching to a lower-precision variant of the model during periods of high KV-…
-
New checksum method enhances CNN fault detection on edge devices
Researchers have developed a new lightweight fault-detection technique called Carry-Through Checksum for convolutional neural networks (CNNs) used in edge applications. This method embeds filters into convolutional laye…
-
Qwen3-8B model scaled for ultra-low-bit language processing
Researchers have successfully scaled post-training ternarisation techniques to the Qwen3-8B language model, aiming to reduce storage and memory requirements. The study involved a comprehensive evaluation, including repr…
-
Embedding table precision is key for LLM size reduction
A recent experiment explored the impact of quantization on Transformer models, revealing that the embedding table constitutes a significant portion (72%) of the model's parameters. The research found that quantizing the…
-
New method enhances 2-bit LLM weight decoding efficiency
Researchers have developed a novel multi-shell decoding method for 2-bit LLM weights, aiming to improve efficiency and quality. The proposed approach includes an offline expansion into GPU layouts and a fused dequantize…
-
LinkedIn unveils new GPU retrieval for semantic search
Researchers have developed a new retrieval framework for semantic search on LinkedIn, aiming to improve the relevance of profile suggestions for users. The system partitions embeddings into eight category-supervised seg…