MK.TOOLS — Robots.txt Generator

Tell crawlers exactly where they can go.

Pick your platform for sensible defaults, fine-tune allow/disallow rules, optionally block AI training crawlers, and download a valid robots.txt — all generated in your browser.

Platform

Upload robots.txt to your web server's root directory (e.g. public_html/robots.txt) so it's served at yoursite.com/robots.txt.

Rules for all crawlers (User-agent: *)

Site URL

Allow paths (one per line, optional)

Disallow paths (one per line)

Crawl-delay in seconds (optional)

Block AI training crawlers

Extra sitemap URLs (optional, one per line)

Custom advanced rules (optional, raw robots.txt syntax)

Live preview

robots.txt

# robots.txt
# Generated with Mark 42's Robots.txt Generator — https://mark42.com/tools/robots-txt-generator

User-agent: *
Allow: /

⚠ Add your site URL to include a Sitemap: reference — this helps crawlers find all your pages.

MK.CTA — Next step

Need your technical SEO handled end to end?

A correct robots.txt is one line item. We audit and fix crawl budget, indexing, site speed, and structure — the whole technical SEO stack.

Book a free strategy call ← All free tools