Ulysse Creation

Table of Contents

WordPress robots.txt: What to Configure — and What Not to Block

Originally published: August 3, 2021 · Last updated: August 13, 2026

The robots.txt file tells compliant crawlers which parts of a site they may or may not crawl. On WordPress, it is useful for crawl management, but it is not a security tool and it is not the right way to remove a page from Google.

How robots.txt works in WordPress

On a standard WordPress installation, visiting https://example.com/robots.txt can return a virtual robots file generated by WordPress. You do not necessarily need a physical robots.txt file in the server root. WordPress exposes filters that plugins and developers can use to alter this output.

A physical file placed in the site root can take precedence, so if an SEO plugin appears to ignore your robots settings, first check whether a real file already exists on the server.

A sensible WordPress robots.txt

For most public WordPress sites, the safest policy is deliberately simple:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemaps.xml

The exact sitemap URL depends on your SEO setup. WordPress core, SEOPress and other SEO plugins can expose different sitemap locations, so use the URL that actually works on your site.

Do not block CSS, JavaScript or images without a reason

Search engines render pages. Blocking front-end assets can make it harder for a crawler to understand the page as users see it. Avoid old recipes that indiscriminately disallow /wp-content/, /wp-includes/, theme folders, plugin folders or image directories.

There are legitimate exceptions, but every rule should have a clear purpose. A shorter robots file is usually easier to audit and less likely to cause accidental deindexing problems.

robots.txt does not mean noindex

This distinction is critical. A Disallow rule controls crawling; it does not guarantee that the URL disappears from search results. If other pages link to a blocked URL, a search engine can still know that the URL exists.

For a page that should not appear in search, use a noindex directive while allowing the crawler to access the page long enough to see that directive. For private or sensitive material, use authentication or access control. Never rely on robots.txt to protect confidential information.

Check the WordPress Search Engine Visibility setting

Under Settings → Reading, WordPress includes a “Discourage search engines from indexing this site” option. This is useful for staging environments, but it should normally be disabled on a production site that you want indexed. Modern WordPress handles this primarily through robots meta directives rather than a blanket Disallow: /.

How to audit your robots.txt

  1. Open /robots.txt in a private browser window.
  2. Confirm that the file is accessible with a normal HTTP response.
  3. Check every Disallow rule and ask what problem it solves.
  4. Make sure important CSS, JavaScript and image resources are crawlable.
  5. Confirm that the sitemap URL listed in the file actually resolves.
  6. After changes, monitor Search Console for crawling or indexing anomalies.

The practical rule

Use robots.txt to manage crawling, not to hide content. On a normal WordPress site, start with the smallest possible set of restrictions and add rules only when you can explain exactly why they are necessary.

Official references: WordPress do_robots() and WordPress SEO documentation.