New hardware extension Ventaglio boosts Transformer inference speed

Researchers have developed a novel hardware extension called Ventaglio, designed to significantly accelerate sparse tensor contractions on vector processors, a critical operation for Transformer model inference. This extension, integrated into a 12nm FinFET cluster, provides native support for indexed gather-accumulate-scatter operations, overcoming limitations in existing RVV architectures. Ventaglio achieves substantial speedups, ranging from 6.9x to 7.4x over optimized baselines, and demonstrates practical performance gains of 2.06x to 5.25x on a DuoGPT-pruned LLaMA-3-8B model with dual sparsity. AI

IMPACT This hardware innovation could lead to more efficient and faster deployment of large language models, reducing inference costs and latency.

RANK_REASON The cluster contains an academic paper detailing a new hardware extension for AI model inference. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• DuoGPT
• FinFET
• Gustavson
• LLaMA-3-8B
• Transformer

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →