PulseAugur
EN
LIVE 12:23:28

llama.cpp's MoE cache boosts Qwen3-30B-A3B speed on Nvidia T4

By PulseAugur Editorial · [1 sources] ·

A user tested llama.cpp's new --moe-cache-mib feature on a 16 GB Nvidia T4 GPU with the Qwen3-30B-A3B model. While the model itself did not fit on the card, the cache feature improved decoding speed from 22.6 to 43.2 tokens/s with a 95.7% hit rate when all experts were in RAM and a 10,000 MiB cache was used. The user noted two issues: the cache details are only printed with a verbose setting, and a minimum-sized cache was slower than having no cache at all. AI

IMPACT Demonstrates potential performance gains for Mixture-of-Experts models with optimized caching on consumer-grade GPUs.

RANK_REASON User testing of a specific software feature on hardware.

Read on Mastodon — mastodon.social →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

llama.cpp's MoE cache boosts Qwen3-30B-A3B speed on Nvidia T4

How we ranked this

Signal score
8 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
User testing of a specific software feature on hardware.
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. Mastodon — mastodon.social TIER_1 English(EN) · sampop3 ·

    We ran llama.cpp's new --moe-cache-mib on a free 16 GB Tesla T4 with Qwen3-30B-A3B, which doesn't fit on the card. With every expert in RAM and a 10,000 MiB cac

    We ran llama.cpp's new --moe-cache-mib on a free 16 GB Tesla T4 with Qwen3-30B-A3B, which doesn't fit on the card. With every expert in RAM and a 10,000 MiB cache, decode went from 22.6 to 43.2 tokens/s at a 95.7% hit rate. Two gotchas: llama-server only prints the cache's lines …

    LINKS inferprice.pages.dev/…/llama-cpp-moe-expe… inferprice.pages.dev/articles

RELATED ENTITIES

RELATED TOPICS