A developer has built a small language model (SLM) from scratch, featuring 216.5 million parameters and a context length of 768 tokens. The model was trained on approximately 551 million tokens from various public English text sources and instruction/chat datasets, utilizing a single NVIDIA RTX 3080 GPU over about 15 hours. The developer is seeking feedback on aspects such as the token budget for pretraining versus SFT, data mix timing, improving factual accuracy, and architectural choices. AI
IMPACT Provides a practical example of building and training a small language model from scratch, offering insights into architectural and data considerations for hobbyists and researchers.
RANK_REASON The item describes a personal project and a small-scale model build, not a release from a major AI lab or significant research.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
