New 3D Architecture ThAME Accelerates LLM Mixture of Experts Inference

Researchers have developed ThAME, a novel 3D heterogeneous multi-chiplet architecture designed to accelerate the inference of Large Language Models (LLMs) that utilize Mixture of Experts (MoE) architectures. ThAME addresses key bottlenecks in MoE inference, including memory bandwidth limitations, non-deterministic traffic from token routing, and synchronous expert output aggregation. The architecture integrates FeFET-based non-volatile memory and DRAM-based volatile memory chiplets with a specialized communication backbone, demonstrating significant improvements in speedup and energy efficiency. AI

IMPACT This new architecture could significantly improve the efficiency and speed of running large, complex LLMs, potentially lowering inference costs and enabling new applications.

RANK_REASON Academic paper detailing a new hardware architecture for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• DRAM
• FeFET
• Large Language Models
• Mixture of Experts
• Pratyush Dhingra
• ThAME

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →