llama-server
PulseAugur coverage of llama-server — every cluster mentioning llama-server across labs, papers, and developer communities, ranked by signal.
12 day(s) with sentiment data
-
Fixing slow LLM responses after restarts with llama.cpp KV cache
A technical explanation addresses the performance degradation experienced with long conversations in local LLM applications like llama.cpp after a server restart. The issue stems from the KV cache, which stores processe…
-
Guide to configuring llama-server for dynamic model loading
This guide explains how to set up llama-server in router mode for dynamic model loading and switching. It details the configuration of models.ini, the setup of a systemd service, and the usage of its API. The article al…
-
MoE Offloading Performance Varies on Consumer GPUs
A local language model enthusiast explored the performance of Mixture-of-Experts (MoE) models on a 16GB VRAM GPU, comparing traditional layer offloading with a newer MoE-specific offloading technique. The experiment inv…
-
Ollama releases v0.40.13-rc1 with LFM3 GGUF fixes and llama-server speech
Ollama has released version v0.40.13-rc1, incorporating fixes for LFM3 GGUF and improvements to its llama-server speech capabilities. This update also includes a version bump for llama.cpp to b11963. The release was man…
-
OpenJet engine enables 4 coding agents on one RTX 3090 at 290 tok/s
A new custom CUDA engine called OpenJet has been developed to run multiple coding agents concurrently on a single GPU, specifically an RTX 3090. This engine allows for a total throughput of approximately 290 tokens per …
-
AI agents in .NET can cut token costs by optimizing tool lists
This article explores methods for reducing token costs associated with AI agents in .NET applications. The author conducted experiments using llama.cpp and a Qwen2 tokenizer to measure the actual token usage of differen…
-
New CLI tool ggufit helps estimate local LLM VRAM needs
A developer created a one-file Python CLI tool called ggufit to help users estimate VRAM requirements for running local LLMs. The tool analyzes .gguf model files or provided parameters to calculate necessary VRAM, accou…
-
Ollama v0.40.7 adds Perplexity's pplx-decider model, improves tool call handling
Ollama has released version 0.40.7, introducing a new decision model called pplx-decider from Perplexity, which is fine-tuned from Qwen3.8-27B. This update also includes fixes for conversation continuity across differen…
-
llama-server API fails to flag truncated replies, risking agent errors
The llama-server's Responses API incorrectly marks truncated replies and tool calls as completed, unlike the chat completions endpoint which correctly identifies length-based cutoffs. This behavior can lead clients to t…
-
AI agents struggle with subjective tasks; llama-server adds OpenAI compatibility
The llama-server project has introduced an OpenAI-compatible endpoint for its responses, though it currently does not report when a reply is cut off. Separately, a discussion explores the challenges of delegating tasks …
-
llama-server bug caused long request delays, now fixed
A bug in the llama-server's HTTP handling, specifically within the cpp-httplib library it uses, could cause significant delays for some user requests. When a large number of keep-alive connections were open, a new conne…
-
llama-server API fails to apply tool_choice and stop_sequences correctly
The llama-server project has identified two significant issues when using its Anthropic-compatible API. Firstly, the `tool_choice` parameter, intended to guide model function calls, is not being correctly applied, leadi…
-
Qwen3.5-122B model powers household finance AI on Mac Studio
A developer details how they integrated the Qwen3.5-122B model into a personal household finance analysis tool called Gharasathi, running it on a Mac Studio. The setup utilizes llama.cpp for direct control over model pa…
-
Bees AI clarifies local model network traffic, assures data privacy
The Bees AI application, designed to run local language models, has detailed its network communications to assure users that prompts and data remain on their local machines. While the core model execution is isolated, t…
-
LiquidAI's d1-omni-600M multimodal model runs on Raspberry Pi
LiquidAI has released the d1-omni-600M, a multimodal model designed for edge devices like the Raspberry Pi. Unlike traditional LLMs, this model cannot generate text; instead, it functions as a sophisticated classifier, …
-
llama.cpp enables local LLM hosting with OpenAI-compatible API
The llama.cpp project has released new tools for running large language models (LLMs) locally. Users can now install llama.cpp to execute GGUF models using the llama-cli command-line interface. Additionally, the project…
-
User details setup for running private AI on a mid-range smartphone
A user details a setup for running a private AI model on a mid-range smartphone, the Samsung Galaxy S23 FE, without relying on cloud services. The system utilizes Termux for installing llama.cpp and a llama-server to ho…
-
Small LLMs struggle with valid JSON for tool calls, requiring new parsing strategies
Local LLM deployments using smaller models, such as 7B parameter models, often struggle with generating valid JSON for tool calls. Common issues include markdown fences, surrounding prose, double encoding, truncation, a…
-
Llama-server prompt cache bug causes incorrect LoRA scale inference
A bug in the `llama-server`'s RAM prompt cache can lead to incorrect text generation when using LoRA adapters at different scales. The cache stores computed KV states but does not record the specific LoRA adapter and sc…
-
Llama-server bug misreports logprobs with speculative decoding
A bug in the llama-server software causes speculative decoding to incorrectly report log probabilities as zero for generated tokens. This issue affects various models and configurations, including Gemma 3B, when specula…