PulseAugur
EN
LIVE 22:16:36

Naver Blog content excluded from AI answers due to robots.txt

By PulseAugur Editorial · [1 sources] ·

A study by Daehyeon Nam, founder of Trina Inc., found that Naver Blog content, despite its high visibility in Google search results for Korean queries, is almost entirely absent from AI-generated answers. This exclusion is attributed not to content quality but to specific directives in Naver Blog's robots.txt file, which explicitly prohibits crawling by AI training and retrieval bots like GPTBot and Google-Extended. The research indicates that blocking these AI crawlers significantly reduces a domain's likelihood of being cited in AI responses, with blocked domains converting search results to citations at approximately one-fifth the rate of unblocked ones. AI

IMPACT Website owners must carefully configure robots.txt to control AI crawler access for content visibility in AI-generated responses.

RANK_REASON Research paper detailing findings on AI crawler behavior and website indexing. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

Naver Blog content excluded from AI answers due to robots.txt

How we ranked this

Signal score
26 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
Research paper detailing findings on AI crawler behavior and website indexing. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
product, policy
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 English(EN) · Daehyeon Nam (남대현) ·

    Naver Blog: 128 Google results, 0 AI citations. We collected 2,435 citations to find out why.

    <blockquote> <p><strong>Disclosure:</strong> I'm Daehyeon Nam (남대현), founder and CEO of <strong>Trina Inc.</strong> (주식회사 트리나), a four-person company in Gyeongju, South Korea that builds GEO (Generative Engine Optimization) agents. This post describes a study we ran ourselves, an…

RELATED ENTITIES

RELATED TOPICS