A high school student has developed a novel method to run a 27B parameter LLM locally on two older Android phones by clustering them. This setup bypasses RAM limitations using a custom C++ engine that splits the model layers via RPC over a USB cable. The system achieves approximately 2.8 tokens/second generation speed without overheating, utilizing a unique ternary quantization approach with runtime Hadamard transforms on the CPU. AI
IMPACT Demonstrates a novel, low-power approach to running large language models on edge devices, potentially enabling wider local AI deployment.
RANK_REASON The item describes a novel technical approach to running an LLM on consumer hardware, including a custom engine and unique quantization method, presented by an individual researcher. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
