1700-1800 : in the University of Illinois Library

Speculative decoding can slow LLMs if acceptance rate is too low

Speculative decoding, a technique intended to speed up large language model inference, can paradoxically slow down performance if not configured correctly. The method involves a smaller "draft" model generating candidat…