1-bit and 1.58 bit LLM Benchmarking on Jetson Orin Nano Super | Bonsai LM
A recent benchmark tested Bonsai LM's 1-bit and 1.58-bit models, ranging from 1.7B to 8B parameters, on a Jetson Orin Nano Super. The tests focused on performance across different power modes (7W, 15W, 25W, MAXN) using llama.cpp CUDA. Key findings indicate that 25W is optimal for models up to 4B parameters for energy efficiency, while 15W is a more power-conservative choice for the 8B model. No thermal throttling was observed, with peak temperatures well below hardware limits. AI
IMPACT Provides insights into efficient LLM deployment on edge devices, informing operators about performance trade-offs.