NInfer
PulseAugur coverage of NInfer — every cluster mentioning NInfer across labs, papers, and developer communities, ranked by signal.
6 day(s) with sentiment data
-
NInfer fork boosts LLM context to 555k with 4-bit KV cache
A fork of the NInfer project has been developed, introducing significant improvements to context length and memory management for large language models. This fork features a custom 4-bit KV cache that reduces VRAM usage…
-
Local LLM generator learns to self-correct code based on video output
A developer has enhanced their "auto_demo_scener" project, which uses local LLMs to generate demoscene effects. The latest update integrates Ninfer support and a new feature allowing the LLM to review its own generated …
-
NInfer, llama.cpp, vLLM speed and quality compared for Qwen3.8-27B
A user conducted a performance comparison of three inference engines—NInfer, llama.cpp, and vLLM—on a single RTX 5090 GPU using the Qwen3.8-27B model. The evaluation focused on quality and speed for a production content…
-
NInfer fork enables 1M context for Qwen-3.8 27B on dual 5090s
A fork of the NInfer C++20/CUDA inference engine has been developed to support a 1 million token context window for the Qwen-3.8 27B model. This enhanced version, running on dual 5090 GPUs, achieves significantly higher…
-
NInfer boosts local LLM performance to 220 tokens/sec on 5090 GPU
A user on Reddit's r/LocalLLaMA subreddit shared their positive experience using NInfer with a 5090 GPU to run a 27 billion parameter model. They reported achieving significantly higher throughput, with speeds averaging…
-
NInfer fork enables 2x performance boost for Qwen3.6-35B on CMP170HX hardware
A user has successfully forked the NInfer project to enable it to run on CMP170HX hardware, achieving a twofold performance increase for the Qwen3.6-35B model. This modification involved adjusting CUDA kernels and compi…
-
Qwen model made more concise with "Sharp" prompt overlay for NInfer
A user has developed a method to make the Qwen model more concise by applying a "Sharp" system prompt overlay to the NInfer inference engine. This modification reduces output tokens by 42% without impacting speed or cor…
-
Qwen3.8-27B model hits 880 tok/s on single RTX 5090 with NInfer engine
A user has achieved impressive performance with the Qwen3.8-27B model on a single RTX 5090 GPU, reaching 880 tokens/second with 4-bit NVFP4 quantization and a full 262k context. This speed was attained using the NInfer …
-
NInfer update enables 350K token context for Qwen 3.8-27B on RTX 4090
A user has updated their fork of NInfer, a tool for running large language models locally, to support the Qwen 3.8-27B model. This update allows for context windows of up to 250-350K tokens on a single RTX 4090 GPU, wit…
-
NInfer adds day-zero support for Qwen3.8-27B model with engine improvements
The NInfer inference engine has released day-zero support for the new Qwen3.8-27B model, enabling local deployment and experimentation. This update brings significant engine improvements, including support for up to 8 c…
-
NInfer engine achieves 542 tok/s for Qwen3.6 on single RTX 5090
A new inference engine called NInfer, built from scratch in C++/CUDA, has been open-sourced, demonstrating impressive performance with the Qwen3.6-35B-A3B model. The engine achieved a sustained speed of 542 tokens per s…