Common Crawl
PulseAugur coverage of Common Crawl — every cluster mentioning Common Crawl across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
7B model ZGCM-1 prioritizes tool use and large context over memorization
Researchers from Zhongguancun Academy and Zhongguancun Institute of AI have developed ZGCM-1, a 7.39B parameter model that prioritizes tool use and a large context window over memorizing vast datasets. This approach all…
-
Data Scout method improves AI pretraining corpus creation
Researchers have developed a new method called Data Scout for creating specialized pretraining corpora for AI models. Unlike traditional approaches that filter large web archives, Data Scout directs targeted crawls base…
-
Anthropic sued for billions over alleged AI training data piracy · 1 source tracked
Anthropic faces a significant lawsuit from Sony Music Publishing and Warner Chappell, who allege the company's AI models were trained on tens of thousands of copyrighted songs without permission. The plaintiffs are seek…
-
New video pretraining method and 10M-hour dataset released
Researchers have introduced LeVJEPA, a novel video pretraining method that significantly reduces computational costs while maintaining or improving downstream accuracy. This approach bypasses common heuristics like arch…
-
New GEO strategy aims to boost content visibility in AI search engines
The article outlines a strategy called Generative Engine Optimization (GEO) for content creators to improve their visibility within AI-powered search engines like ChatGPT and Perplexity. Key recommendations include conf…
-
Study: Over 35% of web pages post-ChatGPT show AI authorship · 9 sources tracked
A recent study by the Pew Research Center indicates that over a third of web pages published since the launch of ChatGPT show signs of AI authorship or substantial editing. This trend is particularly pronounced in .com …
-
New SCRIBES framework extracts web data using reusable scripts
Researchers have developed SCRIBES, a reinforcement learning framework designed to extract structured information from semi-structured web content like HTML tables and lists. This method generates reusable extraction sc…
-
Paper reveals unit bias in LLM corpus statistics, losing significant text data
A new paper highlights a significant discrepancy in how large language model training datasets are measured, specifically concerning web-PDF corpora. Researchers found that statistics often report corpus size in tokens …
-
Bengali LLM support lags due to data scarcity, not demand
Despite having a large number of native speakers, Bengali language support in large language models lags significantly behind languages with fewer speakers. This disparity is not due to a lack of demand but rather a sca…
-
New method audits Chinese web corpora for LLM pollution
Researchers have developed a new method called Sampled-BPE to efficiently audit large Chinese web corpora for language model pollution. This technique significantly reduces runtime and memory usage compared to full scan…
-
AWS Open Data Registry Gets MCP Server for Improved Data Discovery
The Registry of Open Data on AWS, a catalog of over a thousand datasets, faces an interface problem where finding specific data is challenging despite its availability. A new MCP server aims to address this by acting as…
-
New CuraWeb corpus boosts LLM performance with optimized data curation
Researchers have developed CuraWeb, a new 2 trillion token English corpus designed to improve the pretraining data for large language models. Unlike previous methods that focused on singular optimization objectives, Cur…
-
LLMs use em-dashes due to WordPress training data, not inherent preference
Large Language Models (LLMs) may exhibit a tendency to use em-dashes not because they inherently prefer them, but because their training data includes a significant amount of content from WordPress websites. WordPress h…
-
Solo dev builds free backlink checker with Claude MCP integration
A solo developer has created a free backlink checker tool that integrates with Claude's MCP (Model Context Protocol). This allows users to query backlink data, such as domain profiles and referring domains, directly thr…
-
New framework analyzes web crawl data with discovery curves
Researchers have developed a new framework for analyzing longitudinal web crawls, which are sequential samples of an evolving URL population. This framework introduces the "discovery curve" to measure the cumulative URL…
-
CrawlGraph launches free backlink API tier for AI agents
CrawlGraph has introduced a free tier for its backlink API, offering 15 monthly calls using Common Crawl's open hyperlink graph data. This new tier allows users to access backlink profiles for approximately 120 million …
-
Stanford releases 152B-token dataset for financial LLM training
Researchers have introduced the Stanford EDGAR Filings Dataset (SEFD), a new open-source corpus designed to provide clean, long-context documents for training large language models, particularly in the financial domain.…
-
Pokemon Go data used for drone navigation, including military
Niantic's geospatial model, initially trained using data from Pokémon Go player scans of Pokéstops, is reportedly being used for drone navigation, including for military applications. While Niantic stated that only earl…
-
Web graph structure guides language model pretraining data selection
Researchers have developed a new method called WebGraphMix for selecting pretraining data for language models. This approach leverages the web graph's structure to identify central and peripheral documents, hypothesizin…
-
AI transparency debate: 'Open weights' insufficient, requires data and value insight
The article "Open Weights, Closed Minds: What AI Transparency Actually Requires" argues that releasing only model weights, a practice termed "open weights," is insufficient for true AI transparency. While this allows us…