Common Crawl
PulseAugur coverage of Common Crawl — every cluster mentioning Common Crawl across labs, papers, and developer communities, ranked by signal.
5 day(s) with sentiment data
-
New method audits Chinese web corpora for LLM pollution
Researchers have developed a new method called Sampled-BPE to efficiently audit large Chinese web corpora for language model pollution. This technique significantly reduces runtime and memory usage compared to full scan…
-
AWS Open Data Registry Gets MCP Server for Improved Data Discovery
The Registry of Open Data on AWS, a catalog of over a thousand datasets, faces an interface problem where finding specific data is challenging despite its availability. A new MCP server aims to address this by acting as…
-
New CuraWeb corpus boosts LLM performance with optimized data curation
Researchers have developed CuraWeb, a new 2 trillion token English corpus designed to improve the pretraining data for large language models. Unlike previous methods that focused on singular optimization objectives, Cur…
-
LLMs use em-dashes due to WordPress training data, not inherent preference
Large Language Models (LLMs) may exhibit a tendency to use em-dashes not because they inherently prefer them, but because their training data includes a significant amount of content from WordPress websites. WordPress h…
-
Solo dev builds free backlink checker with Claude MCP integration
A solo developer has created a free backlink checker tool that integrates with Claude's MCP (Model Context Protocol). This allows users to query backlink data, such as domain profiles and referring domains, directly thr…
-
New framework analyzes web crawl data with discovery curves
Researchers have developed a new framework for analyzing longitudinal web crawls, which are sequential samples of an evolving URL population. This framework introduces the "discovery curve" to measure the cumulative URL…
-
CrawlGraph launches free backlink API tier for AI agents
CrawlGraph has introduced a free tier for its backlink API, offering 15 monthly calls using Common Crawl's open hyperlink graph data. This new tier allows users to access backlink profiles for approximately 120 million …
-
Stanford releases 152B-token dataset for financial LLM training
Researchers have introduced the Stanford EDGAR Filings Dataset (SEFD), a new open-source corpus designed to provide clean, long-context documents for training large language models, particularly in the financial domain.…
-
Pokemon Go data used for drone navigation, including military
Niantic's geospatial model, initially trained using data from Pokémon Go player scans of Pokéstops, is reportedly being used for drone navigation, including for military applications. While Niantic stated that only earl…
-
Web graph structure guides language model pretraining data selection
Researchers have developed a new method called WebGraphMix for selecting pretraining data for language models. This approach leverages the web graph's structure to identify central and peripheral documents, hypothesizin…
-
AI transparency debate: 'Open weights' insufficient, requires data and value insight
The article "Open Weights, Closed Minds: What AI Transparency Actually Requires" argues that releasing only model weights, a practice termed "open weights," is insufficient for true AI transparency. While this allows us…
-
Microsoft AI data center plans face protests, model data questioned
Microsoft's Build 2026 conference saw protests regarding its AI data center plans, highlighting concerns over power consumption, water usage, and community approval. Concurrently, the company's MAI-Thinking-1 model is u…
-
Microsoft MAI models trained on unlicensed web data
Microsoft has reportedly trained its MAI models using unlicensed web data, contradicting its public claims of using only "enterprise grade, clean and commercially licensed data." The company's approach mirrors that of o…
-
AI Crawler Checker parses robots.txt for 10 major AI bots
A new tool called the AI Crawler Checker has been developed to analyze how major AI crawlers interact with a website's robots.txt file. This tool identifies whether specific AI bots, such as OpenAI's GPTBot or Google's …
-
Language model filters cause epistemic injustice, study finds
A new research paper published on arXiv details how pretraining filters and guardrails in language models can lead to epistemic injustice. The audit found that these systems disproportionately flag content related to ma…
-
New Japanese image-text dataset boosts AI cultural understanding
Researchers have introduced WAON, a large-scale Japanese image-text dataset comprising approximately 155 million examples sourced from native Japanese web content. This dataset aims to improve the cultural understanding…
-
Claude Code automates competitor backlink analysis with new agent
A developer has created a method to find competitor backlinks using Claude Code, an AI assistant. This process automates the tedious task of searching for websites that link to competitors but not to one's own site. The…
-
Developer integrates backlink API with AI for SEO gap analysis
A developer has created a new tool that integrates a backlink API with an MCP (Model-Centric Programming) server, allowing for SEO gap analysis directly within AI models like Claude. This setup enables users to describe…
-
AI Bots Ignore Robots.txt, Attempt Database Scans
Several AI-driven web crawlers, including those from Anthropic's Claude and OpenAI's GPT bot, have been observed ignoring robots.txt directives and attempting to scan databases. These bots, along with others from Baidu,…
-
Infini-News offers fast search for 1.3B news articles
Researchers have developed Infini-News, a toolkit and index designed to provide efficient access to over 1.3 billion news articles from the Common Crawl archive. This new resource includes cleaned text, structured metad…