A maker has successfully built a custom cluster of seven ESP32-S3 microcontrollers to run a large language model with approximately 386 million parameters. This was achieved by dividing the model's layers across the microcontrollers and employing aggressive quantization techniques, including a novel 1.58-bit ternary quantization for weights. The system, detailed on GitHub, uses a wired SPI connection for inter-node communication to reduce latency compared to wireless methods. While the inference speed is slow, generating one word in about nine seconds, the project serves as a proof of concept for running LLMs on resource-constrained hardware. AI
IMPACT Demonstrates feasibility of running LLMs on low-power, distributed hardware, potentially enabling edge AI applications.
RANK_REASON This is a custom hardware project demonstrating LLM inference on microcontrollers, not a frontier model release or significant industry event.
Read on Mastodon — mastodon.social →
- Arduino
- BitNet
- ESP32-S3
- ESP-IDF
- ESP-Now
- GitHub
- Low Zi Hong
- MIT
- Simulations Publications, Inc.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
