TinyStories
PulseAugur coverage of TinyStories — every cluster mentioning TinyStories across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
28.9M parameter LLM runs on $8 microcontroller using Gemma-inspired technique
A 28.9 million parameter language model has been successfully run on an $8 ESP32-S3 microcontroller, achieving approximately 9 tokens per second. This significant advancement is made possible by adapting Google's Per-La…
-
New SV-Attention Offers Certified Selection and Exact Unlearning for AI Models
Researchers have introduced Support Vector Attention (SV-Attention), a novel memory mechanism for AI models that leverages a max-margin approach derived from support vector machines. This method allows for certified sel…
-
New theory predicts neural scaling laws from language statistics
Researchers have developed a new theory that can quantitatively predict the exponents of neural scaling laws for large language models trained on natural language datasets, particularly in data-limited scenarios. This t…
-
AI Training Manager offers adaptive control for machine learning
Researchers have developed an AI Training Manager, a system designed to oversee and adapt machine learning training processes in real-time. This manager acts as a supervisory controller, auditing telemetry data and maki…
-
New methods adapt transformer positional encodings for graph data
Researchers are exploring the application of Rotary Position Encodings (RoPE), a technique widely used in transformers for large language models and vision transformers, to graph-structured data. One approach, termed Wa…
-
Researchers translate transformer attention heads into executable Python programs
Researchers have developed a novel method to translate the opaque attention mechanisms within transformer language models into executable Python programs. This approach involves analyzing attention matrices from specifi…
-
Study reveals training dynamics in small Llama-style model
A study on a small Llama-style language model trained with a fixed, compute-constrained token budget revealed that endpoint performance alone is insufficient for evaluating efficiency. The research used a quantitative e…
-
WAV v1 enhances Transformer training with multi-resolution residual routing
Researchers have introduced WAV v1, a novel method for improving the training of deep decoder-only Transformers. This technique enhances residual routing by incorporating multi-resolution detail bases, which capture dir…
-
Hobbyist trains small LLM from scratch on 8GB VRAM
A Reddit user successfully trained a small language model from scratch using only 8GB of VRAM. The project, available on GitHub, focused on the TinyStories dataset and explored various training techniques. While the res…
-
Tiny LLM runs on emulated 90s CPU within old RTOS
A developer has successfully run a 260,000-parameter LLM, trained on the TinyStories dataset, within an emulated 1990s CPU environment. This setup operates on an 18-year-old Real-Time Operating System (RTOS) that the de…