PulseAugur
EN
LIVE 19:39:29

Kernel.org battles AI crawlers consuming millions of daily requests · 2 sources tracked

By PulseAugur Editorial · [2 sources] ·

Kernel.org is experiencing a significant surge in AI crawler traffic, with millions of daily requests overwhelming its infrastructure. These bots, some employing Anubis proof-of-work, are consuming substantial resources, with one report indicating 14 CPU cores across five nodes are dedicated to rendering commits for scrapers. Despite efforts to block these crawlers, a large volume still bypasses defenses, leading to billions of duplicate URL requests instead of efficient code cloning. AI

IMPACT AI crawlers are increasingly straining web infrastructure, necessitating new defense mechanisms beyond traditional bot blocking.

RANK_REASON The cluster describes infrastructure strain caused by AI crawlers, not a core AI release or research.

Read on Mastodon — mastodon.social →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

Kernel.org battles AI crawlers consuming millions of daily requests · 2 sources tracked

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
The cluster describes infrastructure strain caused by AI crawlers, not a core AI release or research.
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
26 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

Full methodology in our editorial standards.

COVERAGE [2]

  1. Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] ·

    🐧 Kernel. org Battles AI Crawlers Generating Millions of Daily Requests ｢ Konstantin Ryabitsev, who helps run Kernel.org’s infrastructure, says git.kernel.org g

    🐧 Kernel. org Battles AI Crawlers Generating Millions of Daily Requests ｢ Konstantin Ryabitsev, who helps run Kernel.org’s infrastructure, says git.kernel.org gets about 6 million requests each day for individual commit pages. Anubis proof-of-work blocks about two-thirds, but man…

    LINKS linuxiac.com/kernel-org-battles-ai-crawle…
  2. Mastodon — mastodon.social TIER_1 English(EN) · [email protected] ·

    Creepy crawlies — Konstantin Ryabitsev on AI scrapers hammering git.kernel.org. 14 CPU cores across 5 geo-distributed nodes are constantly rendering commits for

    Creepy crawlies — Konstantin Ryabitsev on AI scrapers hammering git.kernel.org. 14 CPU cores across 5 geo-distributed nodes are constantly rendering commits for scrapers, ~20% of total capacity. Bots now solve Anubis proof-of-work and still pull billions of duplicate commit URLs …

    LINKS people.kernel.org/…/creepy-crawlies people.kernel.org/monsieuricon

RELATED ENTITIES

RELATED TOPICS