nodus/distill

Knowledge distillation techniques for LLMs explained

Knowledge distillation is a technique used to train smaller, more efficient language models (LLMs) by leveraging larger, more capable teacher models. Several methods exist, including matching the teacher's probability d…