27B AI model runs locally on Jetson Orin NX 16GB

A user successfully ran the Bonsai 27B language model locally on a Jetson Orin NX 16GB device, achieving usable performance for single-user applications despite the hardware's limitations. The setup involved a custom build of PrismML's llama.cpp fork to support the model's unique 1-bit quantization format, as standard tools like Ollama did not support it. While English reasoning performed adequately, the model struggled with non-English languages, indicating that the extreme compression impacts multilingual capabilities. AI

IMPACT Demonstrates the feasibility of running large language models on low-power, edge devices, potentially enabling new local AI applications.

RANK_REASON User successfully runs a specific LLM on consumer hardware, detailing the technical challenges and limitations.

Read on r/LocalLLaMA →

• Bonsai 27B
• llama.cpp
• Ollama
• PrismML

• product
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →