New 700M parameter model Shibai-700M-Base trained on 18B tokens

A user named TheOneWhoWill has pre-trained a 700 million parameter language model called Shibai-700M-Base. This model was trained on 18 billion tokens and is optimized for Python and Wikitext, with plans to further train it on docstring-based Python code. The model is designed for next-token prediction rather than chat and is considered significantly better than GPT-2. AI

IMPACT This release offers a new, potentially more efficient model for specific tasks like Python code generation, contributing to the diverse landscape of open-source LLMs.

RANK_REASON The cluster describes the pre-training and release of a new, smaller-scale language model by an individual developer, which falls under research. [lever_c_demoted from research: ic=1 ai=1.0]

Read on r/LocalLLaMA →

• GPT-2
• Hugging Face
• Qwen 3 0.6B
• Qwen 3.5 0.8B
• Shibai-700M-Base
• TheOneWhoWill

• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →