This field report details the successful porting of Google's Gemma-4 models (2B, 4B, and 12B parameters) to AWS Inferentia2 hardware. The process involved overcoming three primary obstacles: mixed attention heads, limitations with existing frameworks like vLLM and optimum-neuron, and Neuron compiler constraints. By directly tracing the Hugging Face forward pass and implementing specific strategies for KV-sharing and tensor parallelism, the author achieved coherent serving of all three model sizes with performance metrics comparable to CPU references. AI
IMPACT Enables more efficient deployment of Google's Gemma models on AWS infrastructure.
RANK_REASON Field report detailing technical porting of existing models to new hardware. [lever_c_demoted from research: ic=1 ai=0.7]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
