Modal 推出 Auto Endpoints 以实现最先进的推理延迟

报道来源 [3]

1. arXiv cs.CL TIER_1 English(EN) · WenHung Lee, Jian-Jia Chen, Xiaolin Lin, Pei-Shuo Wang, Chi-Chih Chang, Chun-Che Yang, Ning-Chi Huang, Grace Li Zhang, Kai-Chiang Wu · 2026-06-25 04:00

   Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

   arXiv:2606.24957v1 Announce Type: new Abstract: While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency. E…

2. arXiv cs.LG TIER_1 English(EN) · Sahil Kadadekar · 2026-06-25 04:00

   Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion

   arXiv:2606.25097v1 Announce Type: new Abstract: Speculative decoding accelerates inference by letting a draft model propose tokens for a target model to verify, raising a concrete safety question: at temperature zero, can draft-side behavior leak into safety-scored outputs? We an…

3. Modal blog TIER_1 English(EN) · 2026-06-24 00:00

   Achieve state-of-the-art inference latencies with speculative decoding

   How Modal and Decagon worked together to cut inference latency - and you can too.