# TrueEval — the licensed medical group and clinical operating platform behind healthcare brands # https://trueeval.com # Last updated: 2026-10-11 User-agent: * Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # /api/ is closed EXCEPT /api/trpc/ and /api/img/, in every group (the longer rule wins). # State guides, intelligence articles and blog posts load their content from # /api/trpc in the browser, and article images come from /api/img. Disallowing all of # /api/ (2026-08-25) meant Googlebot rendered those pages empty or as "Not Found"; # search traffic to them fell from that week. /api responses carry X-Robots-Tag: # noindex, so allowing the fetch does not get API JSON indexed. Restored 2026-10-11. # Point at /sitemap.xml, not /api/sitemap.xml. The handler serves both, but /api/ is # Disallow'd two lines up — advertising a sitemap inside a blocked path is at best # ambiguous to crawlers and is flagged as an error by every validator. /sitemap.xml is # also the path crawlers try first, unprompted. Sitemap: https://trueeval.com/sitemap.xml # AI Agent Discoverability # LLMs.txt — Machine-readable site description for AI agents (discoverable via the # tag in ). Kept as a comment because "LLMs-Txt" is a # non-standard robots directive that strict validators flag as invalid. # See: https://llmstxt.org/ # LLMs-Txt: https://trueeval.com/llms.txt # ══════════════════════════════════════════════════════════ # SEARCH ENGINE CRAWLERS # ══════════════════════════════════════════════════════════ # Google # Per the Robots Exclusion Protocol a crawler obeys the MOST SPECIFIC group that # matches it and ignores every other group, including "User-agent: *". A named # group containing only "Allow: /" therefore does not inherit the wildcard # Disallows above — it overrides them. Until 2026-08-25 that meant /api/ and # /admin/ were fully crawlable by Googlebot, Bingbot and ~65 others, with the # admin app protected only by its noindex tag. Every named group now carries the # Disallows explicitly, plus the two /api/ Allows (see the top). If you add a group, add them to it too. User-agent: Googlebot Allow: / Crawl-delay: 1 Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Googlebot-Image Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Googlebot-Video Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Googlebot-News Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Storebot-Google Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Bing / Microsoft User-agent: Bingbot Allow: / Crawl-delay: 2 Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: MSNBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: BingPreview Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Yahoo User-agent: Slurp Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Yandex User-agent: YandexBot Allow: / Crawl-delay: 2 Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: YandexImages Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # DuckDuckGo User-agent: DuckDuckBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Baidu User-agent: Baiduspider Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Sogou User-agent: Sogou Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Naver User-agent: Yeti Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Ecosia (uses Bing index but has own crawler) User-agent: Ecosia Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Brave Search User-agent: BraveBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Mojeek User-agent: MojeekBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Qwant User-agent: Qwantify Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # ══════════════════════════════════════════════════════════ # AI SEARCH ENGINES & LLM CRAWLERS — ALL WELCOME # ══════════════════════════════════════════════════════════ # OpenAI / ChatGPT Search User-agent: ChatGPT-User Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: OAI-SearchBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: GPTBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Google AI / Gemini User-agent: Google-Extended Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: GoogleOther Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Perplexity AI — PerplexityBot indexes; Perplexity-User fetches pages the user asks about in an answer. User-agent: PerplexityBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Perplexity-User Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Anthropic / Claude — ClaudeBot trains; Claude-SearchBot surfaces pages in Claude web search; # Claude-User fetches a page when a Claude user asks about it. Allow all three for AI-answer visibility. User-agent: Anthropic-AI Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: ClaudeBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Claude-SearchBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Claude-User Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: anthropic-ai Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Apple Intelligence / Siri User-agent: Applebot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Applebot-Extended Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Meta AI User-agent: Meta-ExternalAgent Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Meta-ExternalFetcher Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: FacebookBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Cohere AI User-agent: cohere-ai Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # You.com User-agent: YouBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Bytedance / TikTok AI User-agent: Bytespider Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Common Crawl (feeds many AI models) User-agent: CCBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Diffbot (knowledge graph builder) User-agent: Diffbot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Neeva AI (now part of Snowflake) User-agent: NeevaBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Phind (developer AI search) User-agent: PhindBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Kagi Search User-agent: KagiBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Mistral AI User-agent: MistralBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Amazon / Alexa User-agent: Amazonbot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Hugging Face User-agent: HuggingFaceBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # AI21 Labs User-agent: AI21Bot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # Inflection AI (Pi) User-agent: InflectionBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # xAI / Grok User-agent: GrokBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # DeepSeek User-agent: DeepSeekBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # ══════════════════════════════════════════════════════════ # SOCIAL MEDIA CRAWLERS # ══════════════════════════════════════════════════════════ User-agent: facebookexternalhit Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Twitterbot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: LinkedInBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Slackbot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: WhatsApp Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: TelegramBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Discordbot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Pinterestbot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: Redditbot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ # ══════════════════════════════════════════════════════════ # SEO & ANALYTICS CRAWLERS # ══════════════════════════════════════════════════════════ User-agent: AhrefsBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: SemrushBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: MJ12bot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: DotBot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: rogerbot Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/ User-agent: SimilarWeb Allow: / Disallow: /api/ Allow: /api/trpc/ Allow: /api/img/ Disallow: /admin Disallow: /_core/