Tree Speculative Decoding

Bole system accelerates hybrid-attention LLM inference with tree speculation

Researchers have developed Bole, a new system designed to accelerate inference for hybrid-attention large language models. These models combine full attention with recurrent linear attention to manage long contexts more…