Bingbot is Microsoft Bing’s primary search crawler. It follows Bingbot-specific robots.txt groups and supports Crawl-delay. Blocking a path limits crawling but does not reliably remove an already known URL from Bing, so use page-level noindex or Bing removal tools when exclusion from search results is the real goal.
User-agent: Bingbot Allow: / Disallow: /account/ Disallow: /internal-search/ Crawl-delay: 2 Sitemap: https://example.com/sitemap.xml
Bingbot discovers new and updated pages for Microsoft Bing’s search index. When it requests a host, it checks the /robots.txt file for rules addressed to Bingbot. A specific Bingbot group lets you control Bing crawling without changing the policy for Googlebot, AI training crawlers, or other agents.
Allowing Bingbot does not guarantee that every page will be indexed or ranked. It only permits crawling. Bing still evaluates page quality, duplication, canonical signals, response status, internal links, and other indexing factors.
The safest configuration is usually narrow: keep public content available and exclude only paths that Bing does not need to fetch, such as internal search results, account areas, carts, or duplicate parameter spaces. Put the file at the root of each host, for example https://example.com/robots.txt. A file on the main domain does not automatically govern a separate subdomain.
The example below allows public crawling while excluding two private or low-value sections. Replace the paths with real directories from your site. Because a Bingbot-specific group is used, include every Bingbot rule you intend to apply instead of assuming that a separate User-agent: * group will be merged with it.
Bing documents support for the non-standard Crawl-delay directive. It can reduce request frequency when Bingbot is placing measurable pressure on a server. Do not add a large delay by default: an excessive value can slow discovery of new or changed pages. Start with server logs and Bing Webmaster Tools, then use a modest value only when the crawl load is a real problem.
If a site needs broader crawl-rate management, Bing Webmaster Tools also provides crawl controls. Robots.txt should remain a clear access policy, not a substitute for fixing slow responses, unstable hosting, or wasteful URL generation.
A Disallow rule stops compliant crawling of a matching path, but Bing may still know that a URL exists from links or previous crawls. Microsoft’s guidance distinguishes crawl control from indexing control. If a URL must not appear in search, use an accessible page-level noindex directive, an appropriate HTTP header, authentication, or Bing’s URL removal tools depending on the situation.
Do not block a page in robots.txt and expect Bingbot to read a noindex tag inside that blocked page. The crawler must be able to fetch the response before it can see page-level indexing instructions.
User-Agent strings can be spoofed. When suspicious traffic claims to be Bingbot, verify the source with Bing’s verification tool or reverse-and-forward DNS checks rather than trusting the header alone. After publishing changes, use the robots.txt checker and Bing Webmaster Tools’ robots.txt tester to confirm that important public URLs remain crawlable.
/ when only one directory should be excluded.Yes. Microsoft states that Bingbot checks and honors robots.txt directives, including Bingbot-specific Allow and Disallow rules.
Yes. Bing supports Crawl-delay, but a large value can slow discovery and recrawling. Use it only when crawl load is a measured server problem.
Not reliably. Disallow controls crawling, not guaranteed removal from the index. Use noindex on an accessible page or Bing’s removal tools when search exclusion is required.
Use Bing’s Verify Bingbot tool or perform reverse DNS followed by a matching forward DNS lookup. Do not trust the User-Agent header by itself.
No. Robots.txt is scoped to the exact scheme, host, and port where it is served, so each subdomain needs its own file and policy.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
robots.txt Definition, Syntax, and ExamplesRobots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.
User-Agent in robots.txt: Definition and MatchingIn robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.
Disallow in robots.txt: Meaning and ExamplesDisallow tells the matching crawler not to request URL paths beginning with the specified pattern. Disallow: / blocks the entire host for that group, while an empty Disallow: rule is ignored. When rules conflict, the most specific matching path wins.
Allow in robots.txt: Meaning and PriorityAllow explicitly permits the matching crawler to request a path. It is most useful as a narrow exception inside a broader Disallow rule. The longest matching path wins; if equally specific Allow and Disallow rules conflict, Allow is preferred.