FreeToken enables large MoE models on single workstation GPUs

Researchers from UC Berkeley and UT Austin have developed FreeToken, an edge-native Mixture-of-Experts (MoE) serving engine designed to run large language models on single workstation GPUs. This system addresses the challenge of deploying powerful models like GLM-5.2 (753B parameters) on consumer hardware by treating a personal machine as a unified inference platform. FreeToken optimizes computation and model state mapping across available GPU, CPU, and memory resources, enabling interactive speeds for models that would typically require datacenter-class infrastructure. The engine is open-source and available for local deployment, making advanced AI capabilities more accessible to individual developers and smaller teams, particularly in sensitive industries like healthcare and legal where data privacy is paramount. AI

IMPACT Enables deployment of large MoE models on consumer hardware, potentially lowering costs for developers and small teams.

RANK_REASON The article describes a new serving engine that enables existing models to run on consumer hardware, rather than a new model release or fundamental research breakthrough.

Read on MarkTechPost →

• Anthropic
• DeepSeek-V4 Flash
• FreeToken
• GLM-5.2
• Kimi k3
• Linux
• llama.cpp
• Microsoft Windows
• NVIDIA
• Ollama
• OpenAI
• University of California, Berkeley
• University of Texas at Austin

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →