Large LLMs Run on Android Phone via CPU Streaming

A developer has successfully run large language models, including GPT-OSS-120B, Qwen-30B, and Gemma 26B, on an Android phone using only the CPU and phone flash storage. This was achieved by implementing a streaming technique that loads only the necessary model experts for each token, rather than the entire model into RAM. This method, built on top of llama.cpp, allows for models significantly larger than the available RAM to run with identical output to in-memory execution, albeit at slower speeds of 1-5 tokens per second. AI

IMPACT Enables running larger, more capable LLMs on edge devices with limited resources.

RANK_REASON Demonstrates a novel method for running large models on consumer hardware, which is a tool-level innovation.

Read on r/LocalLLaMA →

• android
• Gemma 26B
• GPT-OSS 120B
• llama.cpp
• Qwen-30B

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →