Terms of Use
The website provides technical information and tools on an as-available basis. You are responsible for reviewing generated rules, protecting private content with real access controls, and using the service lawfully without abuse.
The website provides technical information and tools on an as-available basis. You are responsible for reviewing generated rules, protecting private content with real access controls, and using the service lawfully without abuse.
Last updated: July 15, 2026
These Terms of Use apply when you browse the website or use its generator, checker, crawler references, templates, guides, and related features. If you do not agree with these terms, do not use the service. Mandatory rights that cannot lawfully be excluded remain unaffected.
The service provides general technical information about robots.txt, crawlers, search discovery, AI-related access, and website configuration. It does not provide legal advice, security certification, crawler-provider support, search-engine guarantees, licensing advice, or a promise that any automated client will obey a published rule.
Robots.txt is not authentication or authorization. You are responsible for protecting private, paid, licensed, personal, or confidential material with appropriate server-side controls.
Generated robots.txt text and checker results are starting points for review. Before publication, verify the exact crawler tokens, paths, host, sitemap URL, search impact, and compatibility with your infrastructure. You are responsible for testing changes and maintaining backups of your own configuration.
You may use the public tools for legitimate website administration, research, education, and evaluation. You must not misuse the service to attack, probe, overload, disrupt, bypass access controls, submit secrets or unlawful material, automate abusive request volumes, interfere with other users, or attempt unauthorised access to administration or infrastructure.
The operator may restrict access, rate-limit requests, remove content, or suspend features when reasonably necessary for security, maintenance, abuse prevention, legal compliance, or service integrity.
Crawler names, company names, documentation links, and product descriptions belong to their respective owners. Their use is descriptive and does not imply affiliation, endorsement, or authority to act for those providers. Third-party documentation and crawler behavior can change without notice.
The checker requests public robots.txt files from domains submitted by users. The operator does not control those websites, their availability, content, security, or response accuracy.
The service may be changed, interrupted, limited, or discontinued. Features, crawler records, examples, policies, and these terms may be updated as the project develops. Continued use after an updated version is published constitutes acceptance to the extent permitted by applicable law.
The service is provided on an “as available” basis without a promise of uninterrupted operation, complete accuracy, fitness for a specific purpose, search ranking, crawler compliance, AI citation, or protection from scraping. To the extent permitted by applicable law, the operator is not responsible for indirect, incidental, special, consequential, or business losses arising from use of, inability to use, or reliance on the service.
Use of personal data is described in the Privacy Policy and Cookie Policy. Questions about these terms or reports of misuse should be sent through the active method on the Contact page.
No. It produces a technical starting point. You must review the output against your own site, search goals, access controls, contracts, and applicable requirements.
No crawler compliance is guaranteed. Robots.txt is a protocol that compliant crawlers are requested to honor, while non-compliant clients may ignore it.
You may use it for legitimate evaluation of public robots.txt files, subject to these terms, reasonable request volumes, and applicable law. Do not use the service for abuse or unauthorised probing.
No. Names and documentation references are used descriptively and do not imply affiliation, sponsorship, endorsement, or representation.
Yes. Features, crawler records, examples, and terms may be updated as providers, standards, and the project change.
The website processes the domains submitted to its checker and may retain a one-way IP hash, domain, response status, and summary results for diagnostics and abuse prevention. Optional analytics or advertising operate only when configured.
Cookie PolicyThe website uses essential session and security cookies for core operation. Analytics or advertising cookies are used only if the corresponding services are enabled.
ContactUse the email below for technical questions, corrections, crawler documentation updates or problems with the site.
About this websiteThe website is an independent technical resource that helps site owners understand crawler purposes, generate selective robots.txt rules, test live files, and avoid confusing crawler policy with real access security.