Sitemap and robots.txt Generator
Turn a list of page addresses into a sitemap.xml file, and write a robots.txt file that points to it. Each address is checked before anything is written.
sitemap.xml
robots.txt
How it works
sitemap.xml follows the sitemaps.org protocol. The file starts with an XML declaration and a urlset element in the namespace http://www.sitemaps.org/schemas/sitemap/0.9. Each address becomes one url element with a loc, and a lastmod when you add a date. The date must be a real calendar date in the W3C form YYYY-MM-DD. The characters & < > " and ' are escaped in every value, so an ampersand in a URL is written as & in the file.
Limits. A sitemap file holds at most 50,000 URLs, and at most 52,428,800 bytes (50 MB) uncompressed. Each URL is at most 2048 characters. A longer list must be split into several sitemap files.
URL checks. Each address must be absolute, start with https:// or http://, and contain no spaces. Relative addresses such as /about are refused, because a sitemap needs the full address.
robots.txt follows RFC 9309, the Robots Exclusion Protocol. The file starts with a User-agent line (here *, which means every crawler). Each Allow or Disallow rule is written as you entered it, and its path must start with /. After a blank line, the file gets one Sitemap line for each URL in the list, as the sitemaps.org protocol describes.
robots.txt is a request, not a lock. Crawlers that follow the standard read it, but it does not stop anyone from opening a page. Do not rely on it to hide private content.