Robots.txt Generator

Create a robots.txt file with a visual editor. Add rules for different bots, disallow paths, set crawl delays, and include your sitemap URL.

100% client-side. Your data never leaves your browser.

Bot Rule #1

robots.txt Output

How to install

1. Download or copy the robots.txt content

2. Upload to your website's root directory

3. Verify at yoursite.com/robots.txt

4. Test with Google's Robots Testing Tool

In Laravel

A new Laravel app has public/robots.txt with two lines that allow every crawler:

User-agent: *
Disallow:

The web server sends that file before Laravel runs. To give staging a different robots.txt than production, delete the static file and answer the URL with a route:

use Illuminate\Support\Facades\Route;

Route::get('/robots.txt', function () {
    $rules = app()->isProduction()
        ? "User-agent: *\nDisallow:\n\nSitemap: ".url('/sitemap.xml')
        : "User-agent: *\nDisallow: /";

    return response($rules)->header('Content-Type', 'text/plain');
});

A Disallow rule stops crawling, not indexing (see the first question below). For a staging site, a password is the reliable protection.

Laravel 13.x docs: Basic routing, Responses

Related Tools

Markdown Preview Diff Checker JSON Formatter & Validator Base64 Encoder/Decoder

Multiple Bot Rules

Create separate rules for different search engines and bots. Control Google, Bing, AI crawlers, and social media bots independently.

Quick Presets

One-click presets for WordPress, Laravel, allow-all, block-all, and blocking AI training bots. Start with a preset, then customize as needed.

Sitemap Support

Include one or more sitemap URLs to help search engines discover all your pages. Supports multiple sitemaps for large sites.

Need something custom built?

I build custom Laravel tools, web apps, and SaaS platforms. If you need something beyond off-the-shelf, let's talk.

Contact

Frequently Asked Questions

Does Disallow keep a page out of Google?

No. Google's documentation says robots.txt is not a way to keep a page out of Google. A blocked page can still appear in results, without a description, when other sites link to it. To keep a page out, use a noindex meta tag or an X-Robots-Tag header.

Do not block that page in robots.txt as well. Google must crawl the page to see the noindex, so a page that is both blocked and noindexed can stay in the index.

Does Google use Crawl-delay?

No. Google ignores Crawl-delay, and its documentation lists only user-agent, allow, disallow and sitemap as supported fields. Some other crawlers, such as Bing's, read it. If Googlebot puts too much load on your server, return 503 or 429 for a short time: Google slows down when it gets those codes.