Gemma 4-E2B model efficiently served on single TPU v6e chip

The Google Gemma 4-E2B model, a 2-billion-parameter language model, has been successfully served on a single TPU v6e chip, achieving a throughput of 213 tokens per second for a single user and scaling to approximately 2,200 tokens per second across concurrent streams. This configuration also demonstrated accurate handling of OpenAI-style function calls and basic vision queries. However, attempts to load quantized versions of the Gemma 4 model failed due to unimplemented quantization paths and loader bugs related to layer normalization requirements. AI

IMPACT Demonstrates efficient serving of smaller LLMs on specialized hardware, potentially lowering inference costs for specific applications.

RANK_REASON The item details the technical performance and serving challenges of a specific AI model on particular hardware, which falls under research and development. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Docker
• GCloud
• Gemma 4-E2B
• Hugging Face
• OpenAI
• TPU v6e
• vLLM

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →