Robots.txt Generator

Robots.txt Generator writes a robots.txt file from the user-agent groups, allow and disallow paths and sitemap lines you specify, with one switch that appends a blocking group for the known AI crawlers.

Paths are checked as they are written: one that does not begin with a slash is corrected and flagged, because "admin" is not a path, and a path containing a space is called out since crawlers stop reading the value at the space. A Crawl-delay aimed at Googlebot earns a note that Googlebot ignores it, and a group with no rules is written as an empty "Disallow:" with a warning that this allows everything. The output also tells you when "User-agent: *" combined with "Disallow: /" would block your whole site, and reminds you that this file controls crawling rather than indexing — a blocked URL can still surface in results.

The file is assembled in the page from what you typed, so the admin and staging paths you are trying to keep out of search are not sent to a server in the act of hiding them.

Creating a first robots.txt for a new domain, adding a sitemap line to an existing one, blocking the AI crawlers in a single step, and checking that a hand-written file does not accidentally disallow everything.

Also known as: generate robots.txt · robots.txt creator · block googlebot · ai crawler block robots