A user detailed their process of distilling the DeepSeek V4 Pro model into two versions of Gemma: a 26B parameter MoE model and a 12B parameter dense model. The distillation process, which involved repopulating Natural Questions QA pairs, cost $0.36 for the DeepSeek API calls. The user encountered bugs with Unsloth Studio but eventually managed to train the models on a server with two RTX 3090 GPUs. The 26B model consumed more VRAM and achieved a lower training loss, indicating better knowledge absorption, though the evaluation gap between the two models was small. The 12B model was faster and had higher per-GPU throughput. AI
IMPACT Demonstrates practical techniques for model distillation and fine-tuning, potentially informing future open-source model development.
RANK_REASON User-led research and experimentation with model distillation and fine-tuning. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
