VIDRAFT has developed two distinct serving engines for large language models, addressing separate optimization targets. VKAE is a kernel-level acceleration engine designed to maximize throughput on GPUs, achieving up to 23.4x higher performance and over 10,000 tokens/sec in multi-request scenarios. In contrast, VKUE enables a 34.7B parameter model to run across a wide range of hardware, including CPUs, by optimizing for memory bandwidth rather than raw computational power, making it suitable for regulated or on-premise workloads. AI
IMPACT Offers solutions for optimizing LLM deployment across diverse hardware, potentially reducing infrastructure costs and expanding accessibility.
RANK_REASON This is a product release from a company that is not a frontier AI lab, detailing specific technical optimizations for LLM serving.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
