Researchers have developed a hardware tile for FPGAs that efficiently handles the recurrent state of linear-attention language models. This tile, designed for a 369-million-parameter Gated DeltaNet model on an Agilex 5 FPGA, manages a recurrent state that exceeds the device's on-chip memory by paging it to external memory. The study found that this paging process accounts for only 13% of the decoding time per token, indicating its practicality for edge deployments. The performance bottleneck for this model on the FPGA was identified as the delivery of model weights rather than the state paging. AI
IMPACT This research demonstrates a practical approach to deploying large language models on edge devices with limited memory.
RANK_REASON This is a research paper detailing a novel hardware implementation for language models. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →