# agenticfinancegraph.com robots.txt # Humans, search engines and AI retrieval are all welcome. The product IS the # tape, so the public GET endpoints under /api/ are open to crawlers on purpose. # # Until 2026-09-24 this file said "Disallow: /api/" for every crawler it did not # name, which told answer engines the live tape was off-limits and left them # quoting prose figures that the tape had already superseded. User-agent: * Allow: / Allow: /api/state Allow: /api/series Allow: /api/flags Allow: /api/partners Allow: /api/findings Allow: /api/agents Allow: /api/def Allow: /openapi.json Allow: /llms.txt Allow: /llms-full.txt Allow: /embed.js Allow: /.well-known/ # Keyed CSV export and the POST-only signup endpoint: nothing to crawl. Disallow: /api/export Disallow: /api/subscribe # Retrieval, answer-engine and training crawlers: explicitly allowed. # ShapBot (Parallel Web Systems' search index) is welcome; on 9 Oct 2026 it read the # agent pages 20,000 times in twelve hours, so it is asked to pace itself. The pages # change on the sweep clock and each carries its own as_of. User-agent: ShapBot Allow: / Crawl-delay: 30 Disallow: /api/export Disallow: /api/subscribe User-agent: GPTBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: ChatGPT-User Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: OAI-SearchBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: ClaudeBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Claude-SearchBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Claude-User Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: anthropic-ai Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: PerplexityBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Perplexity-User Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Google-Extended Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Google-CloudVertexBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Googlebot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Googlebot-Image Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Applebot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Applebot-Extended Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Bingbot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: DuckAssistBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Amazonbot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Bytespider Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: CCBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: meta-externalagent Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: FacebookBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: cohere-ai Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: MistralAI-User Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: YouBot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: AI2Bot Allow: / Disallow: /api/export Disallow: /api/subscribe User-agent: Diffbot Allow: / Disallow: /api/export Disallow: /api/subscribe Sitemap: https://agenticfinancegraph.com/sitemap.xml Sitemap: https://agenticfinancegraph.com/sitemap-agents.xml # Agent index # llms.txt: https://agenticfinancegraph.com/llms.txt # llms-full.txt: https://agenticfinancegraph.com/llms-full.txt # OpenAPI: https://agenticfinancegraph.com/openapi.json # Agent card: https://agenticfinancegraph.com/.well-known/agent-card.json # Live tape: https://agenticfinancegraph.com/api/state