# https://spiderbrain.ai
#
# Search and AI crawlers are welcome, including the ones that ground generative
# answers: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web,
# anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended,
# Applebot-Extended, CCBot, cohere-ai, Bytespider, Amazonbot,
# Meta-ExternalAgent and DuckAssistBot are all permitted by the rule below.
#
# Those agents are named here in a comment and deliberately NOT as their own
# robots groups. Under RFC 9309 a crawler obeys only the group matching its own
# token and does NOT inherit from "User-agent: *" (Google: "User agent specific
# groups and global groups (*) are not combined"). This file used to give each of
# those sixteen agents a bare "Allow: /" group, which meant none of them ever
# received the exclusions or the Content-Signal below. One group for everyone is
# both simpler and the only version that actually reaches those crawlers.

User-agent: *
# contentsignals.org. search: indexing and linking. ai-input: retrieval,
# grounding and RAG for generative answers, which is how we get cited, so yes.
# ai-train: model training, reserved as an express reservation of rights under
# Article 4 of EU Directive 2019/790. Reserving training does not restrict
# citation, so ai-input=yes with ai-train=no is deliberate and not a conflict.
# "use" is not part of the three-signal vocabulary; it is retained because it is
# what this domain has been serving and unknown keys are ignored per RFC 9309.
Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference
# Every Disallow is stated BEFORE "Allow: /", and the order is load-bearing.
# Google and Bing resolve conflicts by longest matching path, so order is
# irrelevant to them. Simpler parsers, including Python's stdlib
# urllib.robotparser, return the FIRST matching rule instead: with "Allow: /"
# written first, every exclusion below was silently dead for them. Verified with
# that parser before and after. Longest-match parsers are unaffected by the move.
#
# Next.js App Router serves RSC flight payloads for client-side navigation: 505
# of them across two naming schemes, and the two are byte-identical twins
# (/index.txt and /__next._full.txt are the same 93 KB). They are a crawlable
# near-duplicate of every page on the site, in a format no search or answer
# engine parses. Both wildcard patterns are needed; one alone leaves 375
# crawlable. The trailing dot is load-bearing: "/*__next." cannot match
# "/_next/", which holds the real JS and CSS Googlebot does need to render.
# The two prefix rules repeat the root-level cases without a wildcard, because
# parsers that ignore "*" would otherwise treat the patterns as literal paths
# and match nothing at all.
Disallow: /index.txt
Disallow: /__next.
Disallow: /*index.txt
Disallow: /*__next.
# /r/<code> referral links 302 to / and write an r_landing analytics event on the
# way through, so a crawler following one inflates referral counts.
Disallow: /r/
# GET-able JSON endpoints. Enumerated rather than written as "Disallow: /api/",
# because /api/ itself is an indexable marketing page and a prefix rule would
# deindex it.
Disallow: /api/geo
Disallow: /api/blog-index
Disallow: /api/compat/
Disallow: /api/notifications
Disallow: /api/visitor-ips
Disallow: /healthz
Disallow: /updates/unsubscribe
# Stated last so that under a longest-match parser it loses to every rule above,
# and under a first-match parser it is only reached when nothing above matched.
Allow: /

# Nothing is excluded here for being noindex, and that is deliberate.
#
# A Disallow stops a crawler fetching the page at all, so it never reads the
# noindex, which leaves the URL eligible for URL-only indexing from any inbound
# link: strictly worse than allowing the crawl. Google documents the conflict.
# With ~58 indexable URLs there is no crawl-budget argument on the other side, so
# the remaining noindex routes (/dashboard/, /report/, /welcome/, /Data-Governance/,
# /login/) stay crawlable and rely on their markup.
#
# /v2/ and /indexold/ were the worked example here: two near-duplicate archives of
# the pre-redesign homepage, kept crawlable so their noindex could be read. Both
# routes were DELETED on 2026-09-09 once the redesign shipped, which beats either
# mechanism. /pay-test/ was on that list too and also 404s now.

Sitemap: https://spiderbrain.ai/sitemap.xml
# RSS is an accepted sitemap format (Google accepts RSS 2.0 and Atom 1.0). Declared
# because it is generated from the live published set rather than from the build, so it
# is a second discovery path for posts. It was briefly undeclared on 2026-08-10 because
# HEAD returned 404 while GET returned 200, and Google HEADs a declared sitemap during
# discovery; the worker now answers HEAD, verified, so it is declared again.
Sitemap: https://spiderbrain.ai/feed.xml