A second-year university student built a small, character-level Transformer model in PyTorch from scratch, containing approximately 0.9 million parameters. This model was initially pretrained on Japanese literature excerpts and then fine-tuned to function as a chatbot. The student also implemented a recursive self-improvement (RSI) loop, where the model generates answers, self-grades them, and retrains on the best responses to enhance its performance. AI
IMPACT Demonstrates the foundational principles of LLM development and self-improvement loops for educational purposes.
RANK_REASON The item describes the creation of a small, educational language model from scratch, including its architecture and training process, which falls under research. [lever_c_demoted from research: ic=1 ai=1.0]
- chat_gpt.py
- mini_gpt.py
- PyTorch
- Representational State Transfer
- rsi_gpt.py
- Transformer
- web_ui.py
- ZEN University
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →