# robots.txt for mitswesttriangle.com # Made in the Shade of Durham & Chapel Hill # 2026 split-role AI policy: allow answer bots, block training bots # Last updated: 2026-09-28 # ========================================================= # AI SEARCH + ANSWER ENGINES — ALLOW # (Cite our content live in ChatGPT Search, Claude Search, # Perplexity, Bing Copilot, Apple Spotlight, etc.) # ========================================================= User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Applebot Allow: / User-agent: DuckAssistBot Allow: / User-agent: MistralAI-User Allow: / # ========================================================= # AI TRAINING CRAWLERS — BLOCK # (Separate from answer bots above; blocking does NOT hurt # AI search visibility. Prevents content being used to train # foundation models that compete with us.) # ========================================================= User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Amazonbot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: omgili Disallow: / User-agent: FacebookBot Disallow: / # ========================================================= # SEO SCRAPERS — BLOCK (not used in-house) # ========================================================= User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / # NOTE: DataForSeoBot is deliberately NOT blocked — DataForSEO is our own paid # vendor since Semrush was cancelled 2026-09. Blocking it would make their # OnPage / site-audit API silently return nothing for this site. # ========================================================= # DEFAULT RULES — Googlebot, Bingbot, and all other # user-agents not explicitly listed above. # ========================================================= # START YOAST BLOCK # --------------------------- User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /xmlrpc.php Disallow: /wp-json/ Disallow: /?rest_route= Disallow: /?s= Disallow: /search/ Disallow: /page/*/?s= Disallow: /?p= Disallow: /?attachment_id= Disallow: /?replytocom= Disallow: /*?share= Disallow: /trackback/ Disallow: /*/trackback/ Disallow: /feed/ Disallow: /comments/feed/ Disallow: /*/feed/ Disallow: /*/embed/ # REMOVED 2026-09-28: 'Disallow: /author/'. The author archives were set to # noindex via Yoast, and Google must be able to CRAWL a page to see that tag. # Blocking them meant Google could never learn they were excluded — while # author-sitemap.xml was still submitting them. Same fix as Tampa Bay. Disallow: /cgi-bin/ Sitemap: https://mitswesttriangle.com/sitemap_index.xml # --------------------------- # END YOAST BLOCK