GGUF
PulseAugur coverage of GGUF — every cluster mentioning GGUF across labs, papers, and developer communities, ranked by signal.
- uses llama-cli 90%
- used by llama-cli 90%
- instance of Llama-2-13B-Chat-GGUF 90%
- used by POCKET-Darwin-180B 90%
- authored Salvatore Sanfilippo 90%
- instance of POCKET-Darwin-180B-GGUF 90%
- used by XHToken 80%
- used by vLLM 70%
- used by LM Studio 70%
- uses vLLM 70%
- used by Metal 70%
- used by Microsoft Windows 70%
30 day(s) with sentiment data
-
GPU performance for LLMs: Community data sought to reveal real-world token/sec rates
A blog post on dev.to is seeking community input to determine the actual performance of GPUs when running large language models. The author explains that while spec sheets provide theoretical bandwidth, the real-world p…
-
New methods improve LLM quantization accuracy and efficiency
Researchers have developed new methods for quantizing large language models (LLMs) to reduce their size and improve efficiency for deployment. One approach, GSQ (Gumbel-Softmax Quantization), uses a novel scalar quantiz…
-
Edge device RAM for LLMs: Beyond simple subtraction
Running large language models on edge devices with limited RAM requires careful memory management beyond simply subtracting model file size from total device memory. Factors like the KV cache, operating system overhead,…
-
Local LLM benchmark numbers vary widely due to hardware and measurement differences
Local LLM inference benchmarks can be misleading due to numerous variables, including hardware differences, model file variations, and how different servers measure performance. The author discovered that server-reporte…
-
New C-based inference engine Quantr explores running LLMs without GPUs
A new experimental LLM inference engine named Quantr has been developed in C to explore the intricacies of running large language models without relying on GPUs. The project focuses on quantized model weights, GGUF form…
-
Qwen2.5-7B-Instruct achieves 489.8 tok/s with speculative decoding on RTX 3050
A developer has released an update detailing performance improvements for the Qwen2.5-7B-Instruct model when running on an Nvidia GeForce RTX 3050 GPU. The update focuses on speculative decoding techniques within the ll…
-
Ollama vs llama.cpp: Default settings, not engine, cause speed gap
A performance comparison between Ollama and llama.cpp on the same GGUF model revealed that the initial 15% speed difference was due to default settings rather than the underlying engine. When both applications were conf…
-
Mistral AI releases 'Le Chonk' 1T-parameter model for self-hosting
Mistral AI has released "Le Chonk," a 1 trillion parameter, multimodal model with a 1 million token context window, available for download. The model is offered under a modified MIT license that permits commercial use a…
-
DeepSeek-V4-Flash model runs on 64GB CPU via llama-modchip
A project called llama-modchip has developed a method to run the large 284 billion parameter DeepSeek-V4-Flash model on a standard 64GB CPU system without requiring any GPUs. This is achieved through a custom tensor typ…
-
Darwin-180B GGUF files show name-size mismatch due to large lookup table
An analysis of six GGUF quantized files for the Darwin-180B-RSI-R3 model revealed discrepancies between file names and their actual sizes. The study found that a significant portion of the model's parameters, specifical…
-
Nous Research releases Arete-27B model in GGUF format for local use
Nous Research has released quantized versions of their Arete-27B model in the GGUF format. These quants are available for download via Hugging Face, allowing users to run the model locally on their own hardware. The rel…
-
MAPLE project claims 26B LLM runs 6x faster on 6GB GPU
A new open-source project called MAPLE claims to have optimized a Gemma 4 26B-A4B large language model to run at 45.4 tokens per second on a consumer-grade RTX 4050 laptop GPU with 6 GB of VRAM. This is a significant sp…
-
NVIDIA Dory 4B model runnable via llama.cpp tutorial
This tutorial guides users through running NVIDIA's Dory 4B model using llama.cpp. It details how to build a specific fork of llama.cpp, download the Dory 1.0 4B model in GGUF format, and analyze its architecture. The g…
-
GGUF Export Failures Linked to Insufficient Disk Space
This article discusses a common issue encountered when exporting large language models in the GGUF format, specifically when the available disk space is insufficient for the export process. The author highlights that tr…
-
osFoundry's Local Model simplifies running and tuning open LLMs locally
osFoundry's Local Model is a desktop application designed to manage open-weight language models on a user's own hardware. It integrates model exploration, downloading, fine-tuning, quantization, and evaluation within a …
-
KlangAI releases state-of-the-art Danish ASR model pianissimo-da
KlangAI has released pianissimo-da, a new Danish Automatic Speech Recognition (ASR) model. This model is available in multiple formats including GGUF, ONNX, MLX, and CoreML, and is hosted on Hugging Face. It is describe…
-
New open-source tool simplifies LLM quantization for desktop users
A new open-source desktop workspace called iQuantize has been released, designed to streamline the process of quantizing large language models. This tool supports conversions to GGUF, MLX, and ONNX formats, allowing use…
-
180B MoE model runs on 32GB RAM laptop, SSD is the bottleneck
A 180-billion parameter Mixture of Experts (MoE) model, Darwin-180B-RSI, was successfully run on a laptop with 32GB of RAM, achieving approximately 3 tokens per second using only the CPU. Smaller MoE models, POCKET-26B …
-
Local LLM Inference Booms with New Tools and Hardware Support
The landscape of local LLM inference is rapidly advancing, with tools like Ollama, vLLM, and LM Studio enabling useful open-weight models to run on consumer hardware by 2026. Foundational engines such as llama.cpp are p…
-
Choosing Edge LLM Runtimes: Focus on Artifacts, Not Just Speed
This article discusses the complexities of choosing an edge LLM runtime, emphasizing that direct speed comparisons between different runtimes are often misleading. Instead, it advises developers to first consider the ta…