Infini-News: Efficiently Queryable Access to 1.3 Billion Processed Common Crawl News Articles
Researchers have developed Infini-News, a toolkit and index designed to provide efficient access to over 1.3 billion news articles from the Common Crawl archive. This new resource includes cleaned text, structured metadata, language detection, and geographic attribution for each article. The system utilizes Infini-gram indexes, enabling researchers to search the entire archive for text patterns in under a second, thereby facilitating large-scale media research. AI
IMPACT Lowers the barrier for computational social science and NLP research by providing efficient access to a massive news corpus.