GGUF
PulseAugur coverage of GGUF — every cluster mentioning GGUF across labs, papers, and developer communities, ranked by signal.
17 day(s) with sentiment data
-
Local AI model execution and Thailand's AI landscape discussed
This cluster covers two distinct topics related to AI. The first item details how to run GGUF models locally using tools like Ollama and llama.cpp, with guidance on selecting appropriate quantizations for VRAM. The seco…
-
NobodyWho and Cactus: On-Device LLM Engines Compared
A technical comparison highlights two on-device LLM inference engines, NobodyWho and Cactus, detailing their differences in engine design, model format, hardware acceleration, and licensing. NobodyWho utilizes llama.cpp…
-
llama.cpp releases bring OpenVINO updates, Vulkan, GGUF, and SYCL improvements
The llama.cpp project has released several updates, including version b11024 which features an update to OpenVINO 2026.4 and fixes for various compiler warnings. Other recent releases, such as b11022 and b11020, introdu…
-
VIDRAFT's POCKET-35B LLM hits 1M downloads, enabling on-device AI
VIDRAFT's POCKET-35B, a 35-billion-parameter on-device large language model, has achieved over 1 million downloads on Hugging Face within seven weeks of its release. Designed to run on consumer hardware without a dedica…
-
Ollama releases v0.34.1 with MLX and GGUF model creation updates
Ollama has released version 0.34.1, introducing several key updates. The release makes MLX safetensors "ollama create" functionality no longer experimental and improves memory handling for MLX on Apple Silicon. Addition…
-
New method predicts llama.cpp throughput using GGUF metadata
Researchers have developed a method to predict the single-sequence throughput of llama.cpp, a popular framework for running large language models, using GGUF metadata. This approach employs roofline-shaped predictors wi…
-
Ollama vs. llama.cpp: Choosing Your Local LLM Runtime
The article compares Ollama and llama.cpp as runtimes for local LLM inference, highlighting their distinct operational models. Ollama acts as a managed service, simplifying model management and deployment with stable na…
-
New .NET 10 engine enables local LLM inference without C++ or Python
A new inference engine called Glacier.Inference has been developed for .NET 10 that allows local LLMs to run without C++ or Python dependencies. This engine bypasses traditional CUDA toolkits and native C++ DLLs by inte…
-
Antirez releases DeepSeek 4.1 model in GGUF format
Salvatore Sanfilippo, also known as Antirez, has released a quantized version of the DeepSeek 4.1 model in GGUF format. These files are now available on Hugging Face, with versions for both Q2 and Q4 quantization levels…
-
Open-source multimodal models challenge GPT-4o on cost and performance · 2 sources tracked
Open-source multimodal models are rapidly catching up to GPT-4o in performance and cost-effectiveness, with several models like Alibaba's Qwen2.5-VL and Mistral's Pixtral 12B offering competitive capabilities for tasks …
-
Open-source LLMs rival GPT-3.5 performance with efficient quantization
Several new open-source models, including Llama 3.1 8B, Qwen 2.5 7B/14B, and Mistral Nemo 12B, now rival GPT-3.5's performance on various tasks, particularly in coding and reasoning. These models achieve this efficiency…
-
New GGUF quantization methods improve model performance
A researcher has published a blog post detailing new tensor type layouts for GGUF model quantization. These changes aim to improve model performance across the board, though the author does not claim they represent the …
-
Research reveals silently defective LLM artifacts in public registries
A new research paper highlights significant issues with the integrity of large language model (LLM) artifacts available in public registries. The study found that 1.6% of official artifacts from Ollama and some communit…
-
Flash AI shell author forgoes fine-tuning due to Ollama infrastructure limits
The author of the Flash AI shell decided against fine-tuning their custom Onyx models due to infrastructure limitations. Flash relies on Ollama's cloud-based models for users without powerful local hardware, but this se…
-
LLM Tuning: Chat Templates Matter More Than Quantization
A recent analysis of local Large Language Model (LLM) tuning revealed that chat template configuration has a significantly larger impact on model performance than quantization levels. While quantization (e.g., Q4 vs. Q8…
-
User seeks GLM 5.3 Flash model for 192GB RAM with Antirez/DS4 quantization
A user on the r/LocalLLaMA subreddit is inquiring about the possibility of a GLM 5.3 Flash model in GGUF format, specifically optimized for the Antirez/DS4 quantization method and targeting around 192 GB of RAM. The use…
-
Ollama simplifies local LLM hosting, bypassing cloud subscriptions
Ollama is a tool that simplifies the process of hosting large language models locally on a user's own computer, eliminating the need for cloud-based subscriptions like ChatGPT Plus. It abstracts away the complexities of…
-
llama.cpp b10835 fixes CUDA FlashAttention divergence on NVIDIA GPUs
The llama.cpp project has released build b10835, which addresses a critical bug in its f16 FlashAttention implementation on CUDA backends. This update resolves divergence issues that could lead to instability or errors …
-
Qwen3.8-Flash-Next model released with GSQ-RCO quantization, reducing size and boosting speed
A new version of the Qwen3.8-Flash-Next model, named GSQ-RCO, has been released, offering significantly reduced file sizes while maintaining near-baseline quality. This quantization method cuts the model size from appro…
-
Spark-X2.5 LLM now runs locally with official llama.cpp support
The Spark-X2.5 large language model now officially supports local execution via llama.cpp. This enables users to run the model entirely on their CPU, with a demonstration provided for Microsoft Windows. The release incl…