A user has resolved an issue with the GLM-5.2 NVFP4 model running on four DGX Sparks, achieving approximately 24 tokens per second at a 128K context length. The problem involved a bug in the speculative decoding configuration where the draft model's KV cache and metadata were sharded, but its attention mechanism did not recognize this sharding, leading to incorrect calculations. This bug was fixed by ensuring the `decode_context_parallel_size` was correctly copied during configuration, eliminating the previous trade-off between context length and speed. AI
IMPACT User-level optimization details for a specific model configuration, offering insights into performance tuning.
RANK_REASON User-level technical follow-up on optimizing a specific model configuration.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
