llama.cpp
PulseAugur coverage of llama.cpp — every cluster mentioning llama.cpp across labs, papers, and developer communities, ranked by signal.
- 2026-06-08 research_milestone llama.cpp merged a pull request to optimize KV cache performance for the Gemma-4 model. source
- 2026-06-05 product_launch A SYCL backend has been ported to llama.cpp, offering performance improvements for Intel Arc GPUs. source
- 2026-05-30 product_launch llama.cpp released version b9438, adding custom CSS injection for web UI theming. source
- 2026-05-25 research_milestone A fix is expected for llama.cpp to address split mode tensor crashes. source
- 2026-05-25 product_launch A pull request was submitted to improve checkpoint creation and context handling in llama.cpp. source
- 2026-05-24 product_launch llama.cpp released version b9305 with pre-compiled binaries for multiple platforms. source
- 2026-05-17 research_milestone llama.cpp implements MTP optimizations and prompt decode improvements for faster local AI inference. source
- 2026-05-14 product_launch A performance-optimized fork of llama.cpp was released with new features. source
- 2026-05-12 product_launch llama.cpp project integrates llama-eval tool for model benchmarking. source
30 day(s) with sentiment data
-
User seeks help optimizing MTP in llama.cpp server
A user on Reddit is seeking assistance with implementing the "draft-mtp" (Multi-Turn Prompting) feature in the llama.cpp server. They have downloaded a specific model, Qwen3.6-35B-A3B-MTP-GGUF, and are attempting to run…
-
StepFun 3.7 Flash model shows speed benchmarks on M5 Max chip
A user on Reddit shared benchmarks for the StepFun 3.7 Flash model, running it on an M5 Max chip with 128GB of RAM. The model demonstrated fast and responsive performance with short context windows under 16k tokens. Per…
-
llama.cpp update B9387 enhances AMD ROCm support with MFMA
The llama.cpp project has released an update, B9387, which includes significant improvements for AMD ROCm support. This update specifically enables MFMA (Matrix Multiply-Accumulate) operations, but these are currently r…
-
Mozilla simplifies local AI with single-file Llamafiles
Mozilla has released a new project called Llamafiles, which bundles AI model weights, the llama.cpp runtime, and the necessary software into a single, executable file. This innovation simplifies the process of running A…
-
Qwen3.6-35B model runs 128K context on RTX 3060
A user on Reddit has detailed how to run the Qwen3.6-35B-A3B-APEX model with a 128K context window on an RTX 3060 12GB graphics card. This was achieved by utilizing a fork of llama-cpp with CUDA optimizations from spiri…
-
Reddit user seeks multi-user local LLM setup advice
A user on Reddit's r/LocalLLaMA subreddit is seeking advice on setting up a multi-user local LLM service. They have experimented with vLLM and llama.cpp, using llama-swap as a frontend, but are encountering limitations …
-
Developer builds terminal-native AI tool for project management
A developer created a terminal-native application called qlog to help manage multiple projects and improve productivity by integrating with large language models. Frustrated with existing tools that required significant…
-
Guide shows Mac users how to quantize Gemma 4 with llama.cpp
A guide details how to quantize the Gemma 4 large language model on a Mac using llama.cpp. The process involves cloning the llama.cpp repository, setting up a Python environment with necessary dependencies like PyTorch …
-
Users seek functional Deepseek-v4-Flash quantizations
Users on the r/LocalLLaMA subreddit are seeking functional quantizations of the Deepseek-v4-Flash model. One user shared a Hugging Face link to a Deepseek-V4-Flash-FP4-FP8-GGUF quantization, but reported low quality and…
-
LLM KV cache quant benchmarks: q5/q6 outperform q8/q4
A new benchmark analysis reveals that KV cache quantization levels q5 and q6 offer surprisingly good performance for local LLMs, outperforming the commonly used q8 and q4 quantizations. The research, conducted using a f…
-
Nvidia H100 user seeks advice on llama.cpp vs vLLM for 30-user inference
A user is seeking advice on optimizing inference for a large language model on an Nvidia H100 GPU with 94GB of VRAM. They aim to support up to 30 users, with a focus on a large context window and concurrent usage for co…
-
Local LLM setup autonomously builds and deploys game, outshining commercial models
A user at a local AI developer meetup demonstrated the power of a custom, multi-agent local LLM setup, routing traffic between various models including GLM 5.1, Kimi K2.6, and MiMo v2.5-Pro. This setup, running on a ble…
-
Nvidia releases CUDA 13.3, users test llama.cpp compatibility
Nvidia has released CUDA 13.3, the latest version of its parallel computing platform and programming model. This update is now available for download, with release notes providing detailed information on new features an…
-
User seeks advice on local LLM coding setup with new hardware
A user on the r/LocalLLaMA subreddit is seeking advice on setting up a local coding environment. They have a new PC with an RTX 3090 GPU and an Intel Core i9 Ultra CPU, and 32GB of RAM. The user is asking for recommenda…
-
Ollama v0.30.0, Qwen3.5 35B, and 1-bit AI on WebGPU
Ollama's v0.30.0 pre-release is set to improve llama.cpp interoperability. Separately, a new Qwen3.5 35B model is available in GGUF and GPTQ formats, optimized for local inference on consumer GPUs. Additionally, PrismML…
-
Qwen3.5 model struggles with long context at lower quantization
A user on r/LocalLLaMA is experiencing a significant drop in performance with the Qwen3.5 122B A10B model when its context window exceeds approximately 75-80k tokens. The model begins to hallucinate, forget information,…
-
Harbor v0.4.19 launches local coding agents with integrated LLM gateway
Harbor has released version 0.4.19, introducing enhanced capabilities for launching local agentic coding tools. This update allows users to integrate various local inference backends like vLLM, SGLang, and llama.cpp. Ad…
-
wllama brings GGUF LLMs to browser via WebAssembly and WebGPU
A new tool called wllama enables users to run GGUF large language models directly within their web browser. Leveraging WebAssembly and WebGPU, wllama bypasses typical browser limitations like the 4GB memory constraint a…
-
Llama.cpp users debate parallel client and context size interactions
A user on the r/LocalLLaMA subreddit is seeking clarification on how the `-np` (number of parallel clients) and `-c` (context size) flags interact within the llama.cpp server. They are particularly interested in underst…
-
Rejected llama.cpp PR boosts MoE model speed on Strix Halo
A pull request for llama.cpp, which was denied for inclusion in the main project, offers a performance boost for Mixture of Experts (MoE) models on Strix Halo hardware. This modification, developed by pedapudi, can incr…