New Spectral-LSH method compresses LLM prompts efficiently

Researchers have developed Spectral-LSH, a novel training-free method to compress long prompts for language models, addressing the quadratic scaling issue in prefill attention. This technique approximates attention-kernel operators using Krylov subspace methods and random features, then employs SimHash to group similar tokens into macro-tokens. Evaluations on Mistral-7B-Instruct-v0.3 and Qwen2.5 models demonstrated that Spectral-LSH effectively preserves quality at higher compression ratios (8x and 16x) where simpler chunking methods falter, with adaptive backends combining both approaches. AI

IMPACT This method could significantly reduce inference costs for long-context language models, enabling more efficient processing of lengthy inputs.

RANK_REASON The cluster contains an academic paper detailing a new method for prompt compression in language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Mistral-7B-Instruct-v0.3
• Qwen2.5-14B-Instruct
• Qwen2.5-7B-Instruct
• Spectral-LSH

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →