Gemma 4 models: fp8 format overtakes bf16 for larger sizes on AMD MI300X

A technical guide demonstrates how to serve Google's Gemma 4 models of various sizes, from 2B to 31B parameters, on an AMD MI300X GPU using vLLM. The analysis reveals that for smaller models (2B parameters), bfloat16 (bf16) offers faster performance with a single user, while fp8 only becomes competitive when handling eight or more concurrent requests. However, for models with 12B parameters and larger, fp8 consistently outperforms bf16, showing significant speedups across different request loads and prompt lengths. AI

IMPACT FP8 format shows significant performance gains for larger models, potentially influencing deployment strategies for LLMs on specific hardware.

RANK_REASON Technical guide detailing performance benchmarks of model formats on specific hardware. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• 12b Parameter Model
• AMD MI300X
• bfloat16
• Fp8
• Gemma 31b
• Gemma 4
• Hugging Face
• vLLM

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →