Draft Model Contract for G.P.s

Speculative decoding can slow LLMs if acceptance rate is too low

Speculative decoding, a technique intended to speed up large language model inference, can paradoxically slow down performance if not configured correctly. The method involves a smaller "draft" model generating candidat…