Echo Dot 2 runs local LLM with 28M parameters

A user has successfully implemented a local voice pipeline on an Amazon Echo Dot 2, enabling it to run a 28M parameter LLM. This setup utilizes `llama.cpp` and offline speech recognition on the device's limited hardware, which includes an ARMv7 processor and 512 MB of RAM. The system achieves approximately 7 tokens/s during prompt processing and 4 tokens/s during generation, suitable for simple commands like controlling smart home devices. AI

IMPACT Shows potential for running small LLMs on edge devices for local command processing.

RANK_REASON Demonstrates running an LLM on consumer hardware, which is a tool-level application of AI technology.

Read on r/LocalLLaMA →

• 28M LLM
• ARMv7
• Echo Dot 2
• llama.cpp
• Sherpa-ONNX

• product
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →