# robots.txt for https://www.dunsocial.com # # Policy: everything on this site is public marketing content. We want it crawled, # indexed, cited and summarised, by search engines and by AI assistants alike. # Nothing here is disallowed. # # The named groups below are redundant against the `User-agent: *` default, but they # are deliberate: robots.txt matching picks the MOST SPECIFIC group for a bot, so a # named group is an explicit, auditable "yes" that cannot be broken later by someone # tightening the wildcard group. Every token below is taken from the operator's own # documentation (see the citation on each block); vendors rename these, so re-check # the sources before editing. # --------------------------------------------------------------------------- # Default: allow everything. # --------------------------------------------------------------------------- User-agent: * Allow: / # --------------------------------------------------------------------------- # Search engines. AI answers are composed from the top organic results, so classic # indexing is the foundation. Bing matters twice over: ChatGPT search and Copilot # both lean on Bing's index. # Sources: developers.google.com/search/docs/crawling-indexing/google-common-crawlers # bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 # --------------------------------------------------------------------------- User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / # --------------------------------------------------------------------------- # OpenAI. GPTBot = model training. OAI-SearchBot = the crawler that decides whether # we can appear in ChatGPT search results. ChatGPT-User = a live fetch when a person # asks ChatGPT about us — allowed here, but we serve it HTML (not text/markdown) because # ChatGPT's browser sandbox rejects the markdown MIME. All three must be allowed to cite us. # Source: https://developers.openai.com/api/docs/bots # --------------------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-AdsBot Allow: / # --------------------------------------------------------------------------- # Anthropic. ClaudeBot = training. Claude-SearchBot = Claude's web search index. # Claude-User = a live fetch on behalf of someone talking to Claude. # Source: https://support.claude.com/en/articles/8896518 # (The old `anthropic-ai` and `Claude-Web` tokens are retired and no longer listed # by Anthropic, so they are not repeated here.) # --------------------------------------------------------------------------- User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / # --------------------------------------------------------------------------- # Perplexity. PerplexityBot builds the search index that Perplexity answers from. # Perplexity-User is a live fetch triggered by a user's question. # Source: https://docs.perplexity.ai/guides/bots # --------------------------------------------------------------------------- User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # --------------------------------------------------------------------------- # Google AI. Google-Extended is a training/grounding opt-out token, separate from # Googlebot: blocking it does not affect Search ranking, but it does remove us from # Gemini grounding. We want to be grounded, so it is allowed. # Source: developers.google.com/search/docs/crawling-indexing/google-common-crawlers # --------------------------------------------------------------------------- User-agent: Google-Extended Allow: / User-agent: Google-CloudVertexBot Allow: / # --------------------------------------------------------------------------- # Apple AI. Applebot-Extended is an opt-out-only token (there is no separate crawler # behind it); leaving it allowed lets Apple use crawled content for its AI features. # Source: support.apple.com/en-us/119829 # --------------------------------------------------------------------------- User-agent: Applebot-Extended Allow: / # --------------------------------------------------------------------------- # Meta AI. meta-externalagent trains and grounds Meta's models; meta-webindexer feeds # Meta AI search; meta-externalfetcher is a live, user-triggered fetch. # Source: developers.facebook.com/docs/sharing/webmasters/web-crawlers/ # --------------------------------------------------------------------------- User-agent: meta-externalagent Allow: / User-agent: meta-webindexer Allow: / User-agent: meta-externalfetcher Allow: / # --------------------------------------------------------------------------- # Other AI crawlers and corpora. CCBot (Common Crawl) is upstream of a large share of # open training sets, so a single allow here propagates widely. # Sources: commoncrawl.org/ccbot, developer.amazon.com/amazonbot, # duckduckgo.com/duckduckgo-help-pages/results/duckassistbot # --------------------------------------------------------------------------- User-agent: CCBot Allow: / User-agent: Amazonbot Allow: / User-agent: DuckAssistBot Allow: / User-agent: MistralAI-User Allow: / User-agent: cohere-ai Allow: / # --------------------------------------------------------------------------- # Link unfurlers. Not indexers: these fetch a URL to render the OG card when someone # shares a DunSocial link. A social tool that cannot be previewed on socials is a bad # look, so they are explicitly welcome. # --------------------------------------------------------------------------- User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: Slackbot-LinkExpanding Allow: / User-agent: Discordbot Allow: / Sitemap: https://www.dunsocial.com/sitemap.xml Sitemap: https://www.dunsocial.com/docs/sitemap.xml