Fused Chunked KL Loss

New methods slash LLM distillation costs and boost context length

A new paper from Multiverse Computing introduces two methods to make knowledge distillation for large language models more efficient. The first method, offline distillation, caches the teacher model's top-K logits, redu…