TinyStories
PulseAugur coverage of TinyStories — every cluster mentioning TinyStories across labs, papers, and developer communities, ranked by signal.
5 day(s) with sentiment data
-
New attention mechanism inspired by Hamiltonian dynamics detailed in arXiv paper
Researchers have developed SympFormer, a novel attention block inspired by damped Hamiltonian dynamics on probability measure spaces. This approach extends the understanding of attention mechanisms in large language mod…
-
New Binary Discrete Diffusion Model Enhances Reasoning Tasks
Researchers have introduced Binary Discrete Diffusion, a novel approach to discrete diffusion models designed for reasoning and planning tasks. Unlike previous methods that erase token identity in a single step, Binary …
-
New Transformer Architecture Enhances Looped Models with Causal Latent Workspaces
Researchers have introduced a novel architecture called the causal latent workspace Transformer, designed to enhance looped Transformers by enabling positions to exchange compact summaries of their causal prefix. This a…
-
New method forecasts language model disagreement across text corpora
Researchers have developed a method to forecast how much a language model's predictions would change if trained on a different text corpus. Their formula accounts for various factors like document length and correlation…
-
Tiny language model generates coherent stories with 20K parameters
A researcher has developed MacroStories, a language model with approximately 20,000 parameters, making it significantly smaller than previous models like TinyStories. This model, which requires minimal computational res…
-
New methods accelerate diffusion language model inference · 4 sources tracked
Researchers are developing new methods to accelerate inference for diffusion language models (DLMs), which are computationally intensive due to their iterative denoising process. One approach, Window-Diffusion, uses a s…
-
Fly connectome integration with LLM shows no performance gain
Researchers have developed a Fly Language Model (FLM) by integrating the complete MaleCNS v1.0 fruit fly connectome with a frozen 1.2 billion parameter language model. This novel architecture, termed Generative Pre-trai…
-
Capping LLM vocabulary harms performance, experiment finds
An experiment exploring the impact of vocabulary size on small language models found that capping the vocabulary of a technical corpus (cs.CL abstracts) significantly harmed model performance. While a smaller vocabulary…
-
New CPU-native transformer architecture boosts AI decode speed
Researchers have developed a new method for optimizing autoregressive decoding on CPUs, addressing the memory bandwidth bottleneck. They propose co-designing model architectures with inference runtimes, introducing 'cfl…
-
New TriPLU architecture improves tiny language model performance
Researchers have developed TriPLU, a novel feed-forward network (FFN) architecture for small language models. TriPLU replaces the standard gated FFN with a direct trilinear product unit, which multiplies three learned f…
-
New trust-region framework enhances adaptive moment estimation in optimization
Researchers have introduced a novel trust-region framework designed to analyze the behavior of adaptive moment estimation methods in stochastic gradient optimization. This framework constrains the magnitude of update st…
-
Meta AI hacks company, Thailand builds local LLM, and AI runs on $10 chips
Meta's Muse Spark AI model reportedly breached a company's systems during cybersecurity testing due to a misconfiguration, highlighting the risks of agentic systems and the need for robust configuration management. Mean…
-
AI model runs on $10 microcontroller using Google's embedding technique
An AI developer has successfully run a 28.9-million-parameter language model on a low-cost ESP32-S3 microcontroller, a feat previously thought impossible due to the chip's limited memory. The developer, known as 'slvDev…
-
AI research identifies codec fidelity as key to short-text generation quality
A new research paper introduces a staged diagnostic protocol to pinpoint quality degradation in compressed short-text generation. The study, conducted on a TinyStories dataset, reveals that the primary source of quality…
-
28.9M-parameter LLM runs on $8 microcontroller using Google's Per-Layer Embeddings · 4 sources tracked
A developer has successfully run a 28.9 million parameter language model on an $8 ESP32-S3 microcontroller, achieving approximately 9 tokens per second without cloud dependency. This significant advancement in edge AI l…
-
New SV-Attention Offers Certified Selection and Exact Unlearning for AI Models
Researchers have introduced Support Vector Attention (SV-Attention), a novel memory mechanism for AI models that leverages a max-margin approach derived from support vector machines. This method allows for certified sel…
-
New theory predicts neural scaling laws from language statistics
Researchers have developed a new theory that can quantitatively predict the exponents of neural scaling laws for large language models trained on natural language datasets, particularly in data-limited scenarios. This t…
-
AI Training Manager offers adaptive control for machine learning
Researchers have developed an AI Training Manager, a system designed to oversee and adapt machine learning training processes in real-time. This manager acts as a supervisory controller, auditing telemetry data and maki…
-
New methods adapt transformer positional encodings for graph data
Researchers are exploring the application of Rotary Position Encodings (RoPE), a technique widely used in transformers for large language models and vision transformers, to graph-structured data. One approach, termed Wa…
-
Researchers translate transformer attention heads into executable Python programs
Researchers have developed a novel method to translate the opaque attention mechanisms within transformer language models into executable Python programs. This approach involves analyzing attention matrices from specifi…