A user has successfully trained a 1.1 billion parameter large language model from scratch for approximately $200. The model, named 'gemmeh', was pre-trained on 20 billion tokens from the fineweb-edu dataset and then fine-tuned using LoRA on the openhermes dataset to create a chat-capable model. The user has shared the code, model weights, and a demo website for the project, detailing the architectural choices and training process. AI
IMPACT Demonstrates the feasibility and low cost of training custom LLMs, potentially enabling wider experimentation and development.
RANK_REASON User-generated research project detailing LLM training from scratch. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
