CFGzip

New technique compresses token space to speed up LLM constrained decoding

Researchers have developed CFGzip, a novel offline technique designed to significantly speed up constrained decoding in large language models (LLMs). This method compresses the token search space, drastically reducing t…