AI companies are aggressively crawling open-source infrastructure, causing significant outages and disruptions for projects like SourceHut, KDE GitLab, and GNOME. These AI scrapers often disregard robots.txt and mimic legitimate user agents, making it difficult to implement effective defenses. As a result, some projects have resorted to implementing challenging proof-of-work systems to block these bots, which can also impact legitimate users. AI
IMPACT AI data scraping practices are straining open-source infrastructure, potentially hindering collaboration and development.
RANK_REASON Reports on widespread infrastructure strain caused by AI data scraping practices affecting multiple open-source projects. [lever_c_demoted from significant: ic=1 ai=0.7]
Read on HN — AI infrastructure stories →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
