A user has trained a small language model with 0.5 million parameters on 1 billion tokens from the FineWeb-Edu dataset. This model, named Silia-v2, is an iteration on a previous research paper and incorporates architectural improvements such as Qwen's HydraHead and Apple's Attention Free Transformer to reduce compute requirements. The model has been benchmarked against other small models on HellaSwag, PIQA, and LAMBADA datasets. AI
IMPACT Demonstrates continued experimentation with small-scale models and novel architectural components.
RANK_REASON User-trained model release with associated paper and benchmarks. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
