New C#/.NET engine TensorSharp rivals llama.cpp performance

TensorSharp, a new inference engine for GGUF models, has emerged as a pure C#/.NET alternative to established C/C++ engines like llama.cpp. Developed by zhongkaifu, it offers broad model and backend support, including CUDA, Metal, and Vulkan, and features like paged KV cache and continuous batching. Benchmarks indicate TensorSharp performs comparably to llama.cpp, with some scenarios favoring TensorSharp in speed, while others show it slightly lagging, highlighting the growing maturity of .NET for demanding AI tasks. AI

IMPACT Potentially broadens the accessibility of local LLM inference for .NET developers.

RANK_REASON New software tool release that integrates with existing AI models.

Read on dev.to — LLM tag →

• C#
• CUDA
• Gemma
• GGUF
• llama.cpp
• Metal
• Qwen
• TensorSharp
• vLLM
• Vulkan
• zhongkaifu

• infra
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →