Common Crawl
PulseAugur coverage of Common Crawl — every cluster mentioning Common Crawl across labs, papers, and developer communities, ranked by signal.
9 day(s) with sentiment data
-
Stanford releases 152B-token dataset for financial LLM training
Researchers have introduced the Stanford EDGAR Filings Dataset (SEFD), a new open-source corpus designed to provide clean, long-context documents for training large language models, particularly in the financial domain.…
-
Pokemon Go data used for drone navigation, including military
Niantic's geospatial model, initially trained using data from Pokémon Go player scans of Pokéstops, is reportedly being used for drone navigation, including for military applications. While Niantic stated that only earl…
-
Web graph structure guides language model pretraining data selection
Researchers have developed a new method called WebGraphMix for selecting pretraining data for language models. This approach leverages the web graph's structure to identify central and peripheral documents, hypothesizin…
-
AI transparency debate: 'Open weights' insufficient, requires data and value insight
The article "Open Weights, Closed Minds: What AI Transparency Actually Requires" argues that releasing only model weights, a practice termed "open weights," is insufficient for true AI transparency. While this allows us…
-
Microsoft AI data center plans face protests, model data questioned
Microsoft's Build 2026 conference saw protests regarding its AI data center plans, highlighting concerns over power consumption, water usage, and community approval. Concurrently, the company's MAI-Thinking-1 model is u…
-
Microsoft MAI models trained on unlicensed web data
Microsoft has reportedly trained its MAI models using unlicensed web data, contradicting its public claims of using only "enterprise grade, clean and commercially licensed data." The company's approach mirrors that of o…
-
AI Crawler Checker parses robots.txt for 10 major AI bots
A new tool called the AI Crawler Checker has been developed to analyze how major AI crawlers interact with a website's robots.txt file. This tool identifies whether specific AI bots, such as OpenAI's GPTBot or Google's …
-
Language model filters cause epistemic injustice, study finds
A new research paper published on arXiv details how pretraining filters and guardrails in language models can lead to epistemic injustice. The audit found that these systems disproportionately flag content related to ma…
-
New Japanese image-text dataset boosts AI cultural understanding
Researchers have introduced WAON, a large-scale Japanese image-text dataset comprising approximately 155 million examples sourced from native Japanese web content. This dataset aims to improve the cultural understanding…
-
Claude Code automates competitor backlink analysis with new agent
A developer has created a method to find competitor backlinks using Claude Code, an AI assistant. This process automates the tedious task of searching for websites that link to competitors but not to one's own site. The…
-
Developer integrates backlink API with AI for SEO gap analysis
A developer has created a new tool that integrates a backlink API with an MCP (Model-Centric Programming) server, allowing for SEO gap analysis directly within AI models like Claude. This setup enables users to describe…
-
AI Bots Ignore Robots.txt, Attempt Database Scans
Several AI-driven web crawlers, including those from Anthropic's Claude and OpenAI's GPT bot, have been observed ignoring robots.txt directives and attempting to scan databases. These bots, along with others from Baidu,…
-
Infini-News offers fast search for 1.3B news articles
Researchers have developed Infini-News, a toolkit and index designed to provide efficient access to over 1.3 billion news articles from the Common Crawl archive. This new resource includes cleaned text, structured metad…
-
FutureSim benchmark tests AI forecasting with historical data
Researchers from the Max Planck Institute have introduced FutureSim, a new benchmark designed to evaluate AI agents' ability to predict real-world events using only historical web data. This method prevents agents from …
-
Elsevier sues Meta over AI training data, citing copyright infringement
Academic publishing giant Elsevier, along with other publishers and authors, has filed a lawsuit against Meta, accusing the company of illegally scraping and using copyrighted research papers to train its Llama large la…
-
LLM-generated content is rapidly growing on the web, study finds
A new research paper introduces DeGenTWeb, a system designed to systematically identify websites dominated by content generated by large language models (LLMs) with minimal human oversight. The study found that LLM-domi…
-
News publishers demand Common Crawl block AI training on their content
News publishers are demanding that Common Crawl cease its unauthorized scraping of web content and prevent AI companies from using this data for model training. The News/Media Alliance has formally communicated this dem…
-
Google warns of increasing, unsophisticated AI prompt injection attacks
Google Threat Intelligence researchers have identified an increase in indirect prompt injection attacks targeting AI systems that browse the web. While many of these attacks are currently low in sophistication and harml…
-
Interactive guide explains how large language models like ChatGPT are built
A new interactive visual guide, based on Andrej Karpathy's lecture, explains the intricate process of building large language models. It details the journey from collecting vast amounts of internet text to the final sta…
-
Researchers unveil PermaFrost-Attack for latent LLM poisoning during pretraining
Researchers have introduced PermaFrost-Attack, a novel method for embedding hidden vulnerabilities, termed 'logic landmines,' into large language models during their pretraining phase. This attack, known as Stealth Pret…