Gemma 4 12B struggles with audio attention on large prompts

Users are encountering issues with Google's Gemma 4 12B unified model, which is designed to process audio, vision, and text simultaneously. While the model responds well to audio with short text prompts, it appears to lose its ability to attend to speech when presented with large, dense system prompts. This limitation has been observed across multiple serving frameworks, suggesting a potential issue with the model's architecture or attention mechanisms when handling competing inputs. AI

IMPACT Highlights potential limitations in unified multimodal models when processing long contexts, impacting voice assistant development.

RANK_REASON User-reported issue with a specific model's functionality. [lever_c_demoted from research: ic=1 ai=1.0]

Read on r/LocalLLaMA →

• Gemma 4 12B
• LiteRT-LM
• llama.cpp
• vLLM

• model release
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →