“Discourage Search Engines From Indexing This Site” – What This WordPress Setting Really Does

There’s a single tick box buried in your WordPress settings that can make your entire website invisible on Google and it is one that gets left on after launch more often than you’d think.

It’s called “Discourage search engines from indexing this site”, and it’s one of the most common reasons a brand new website gets no traffic for weeks or months after launch. The site looks perfect. The content is there. But nobody can find it because WordPress has been politely asking Google to stay away the whole time.

The short version: the setting lives at Settings → Reading → Search engine visibility. It should be on while your site is being built, and off the moment it goes live. If your site has been live for months and isn’t showing up on Google, this is the first thing to check.

Here’s what the setting actually does, when you’d genuinely want it on, and how to check yours in about thirty seconds.

What does “discourage search engines from indexing this site” actually do?

When you tick that box, WordPress does two things behind the scenes.

  1. It adds a “noindex” instruction to every page
    WordPress inserts a small line of code into the header of every page telling search engines not to add it to their index:Search engines read that instruction and skip the page.
  2. It rewrites your robots.txt file
    WordPress also changes the file search engines check before crawling your site — yourdomain.com.au/robots.txt — to say:User-agent: *
    Disallow: /That’s the whole site, closed off.

    Think of it like a “no visitors, thanks” note on your front door. Google reads the note and moves on without cataloguing what’s inside your shop.

The important word is discourage. WordPress is making a request, not enforcing a rule. Google and Bing honour it. But it doesn’t physically block anyone – your site is still live, still publicly reachable, and anyone with the link can visit it. Less reputable crawlers and scrapers can, and do, ignore the request entirely.

That distinction matters, and we’ll come back to it.

Where to find the setting in WordPress

  1. Log in to your WordPress dashboard.
  2. Go to Settings → Reading.
  3. Scroll to Search engine visibility near the bottom.
  4. The tick box reads “Discourage search engines from indexing this site”, with a note underneath saying “It is up to search engines to honour this request.”
  5. Tick or untick as needed, then click Save Changes.That’s it. There’s no separate confirmation, and the change takes effect immediately.

4 ways to check whether it’s switched on

You don’t need to be technical to check any of these.

  1. The dashboard notice
    If the setting is on, WordPress usually displays a warning at the top of your dashboard: “Search engine visibility: Search engines are discouraged from indexing this site.” It’s easy to scroll past, but it’s there.
  2. Your robots.txt file
    Type yourdomain.com.au/robots.txt into your browser. If you see Disallow: / on its own line, your whole site is being blocked.
  3. Your SEO plugin
    Yoast, Rank Math and All in One SEO all flag this setting with a prominent warning, usually on their dashboard or in the SEO health check. If you have one installed, it will have been telling you.
  4. Google itself
    Search Google for site:yourdomain.com.au with no space after the colon. That shows you every page of your site Google has indexed. If nothing comes back, or only a couple of pages do, something is stopping it.

When you should have it turned on

There are three legitimate reasons.

  • While your site is being built
    You don’t want Google indexing half-finished pages, lorem ipsum placeholder text, or a “coming soon” holding page. Those pages can linger in search results long after they’ve been replaced, and they make a poor first impression on anyone who finds them.
  • On a staging or test copy
    If you have a duplicate of your site at something like staging.yourbusiness.com.au, you absolutely don’t want it indexed. Two identical sites competing in search results is a genuine SEO problem, and Google may end up ranking the staging version instead of the real one.
  • On a private or internal site
    Client portals, internal resource hubs, members-only sites – anything with no business appearing in search results.

Outside of those three situations, it should be off. Always.

The expensive mistake: forgetting to turn it off

This is the one that costs small businesses real money.

A developer ticks the box on day one of a build. The site goes live six weeks later. Everyone celebrates. Nobody unticks the box.

Three months on, the business owner is wondering why the phone isn’t ringing and why their site doesn’t appear when they Google their own name.

They’ve paid for content, paid for a logo, maybe even paid for SEO work and none of it can do anything, because Google was never invited in.

The damage compounds. Search engines don’t just skip your site once. Visibility and trust in search build slowly over time, so every month spent hidden is a month of ground you have to make up later. A site that could have been ranking for local searches by month three might now be starting from zero.

There’s a second-order cost too. Businesses in this situation often assume the website itself is the problem and spend money redesigning it, or hire an SEO agency to fix rankings that were never possible in the first place.
If you’ve launched a new site in the last year and organic traffic has been flat, check this before you spend another dollar on marketing.

How to turn it off, and what to do next

Unticking the box removes the barrier, but search engines still need to come back and crawl your pages. Here’s the full sequence.

  1. Untick the box. Settings → Reading → Search engine visibility → Save Changes.
  2. Confirm your robots.txt has changed. Visit yourdomain.com.au/robots.txt again. The Disallow: / line should be gone.
  3. Clear your caching plugin. WP Rocket, LiteSpeed Cache, W3 Total Cache and similar will happily keep serving the old version of your pages, complete with the noindex tag.
  4. Set up Google Search Console if you haven’t already. It’s free, and it’s the only tool that tells you what Google can and can’t see on your site.
  5. Submit your sitemap. Most SEO plugins generate one at yourdomain.com.au/sitemap_index.xml or /sitemap.xml. Paste that into Search Console under Sitemaps.
  6. Request indexing for your key pages. Use the URL Inspection tool in Search Console for your homepage, main service pages and contact page.
  7. Claim your Google Business Profile while you’re at it, if you’re a local business. It’s often the faster route to visibility than organic rankings.

How long until you show up on Google?

Google re-crawls your homepage from a few days to 2 weeks.
Most pages are indexed in 2-4 weeks
Appearing for your own business name is usually 1-4 weeks
Ranking for competitive local terms is usually 3-6 months

Requesting indexing in Search Console usually speeds up the first two considerably. Rankings are a separate matter, being indexed just means you’re eligible to rank, not that you will.

Still not appearing? 6 other things to check

If you’ve unticked the box and nothing has changed after a month, the cause is likely one of these.

  • A caching plugin still serving old pages. Clear every cache, including your host’s and any CDN like Cloudflare.
  • A noindex tag set in your SEO plugin. Yoast and Rank Math have their own per-page and site-wide noindex settings, entirely separate from the WordPress one.
  • A manually edited robots.txt file. Some hosts and security plugins write their own, which overrides the WordPress default.
  • A holding or maintenance mode plugin still switched on and blocking crawlers.
  • Your site is brand new. Domains with no history can take a few weeks to be picked up. Patience, plus a Search Console submission.
  • A manual action or security issue. Check Search Console under Security & Manual Actions. Rare, but worth ruling out.

The bottom line

  • The setting lives at Settings → Reading → Search engine visibility.
  • It asks search engines not to index your site, it doesn’t block anybody.
  • Keep it on during a build, on staging sites, and on private sites.
  • Turn it off the day your site goes live, and put it on your launch checklist so it can’t be forgotten.
  • After unticking, clear your cache and submit your sitemap in Google Search Console.

This is one of a handful of small technical things that quietly hold good websites back. They’re usually quick to fix once you know they’re there – the hard part is knowing to look.

Is your website actually visible on Google?

If you’re not sure whether your site is being found, or you’ve had a nagging feeling that it should be performing better, let us take a look. Design, speed, SEO and the technical settings most people never check.

FREQUENTLY ASKED QUESTIONS

Settings → Reading → Search engine visibility, near the bottom of the page. Tick or untick the box and click Save Changes.

No. Your site stays completely public and anyone with the URL can view it. It only asks search engines not to list it in results. For genuine privacy you need password protection or a login requirement.

Google and Bing do honour it. Many smaller crawlers, scrapers and AI bots don’t, because it’s a request rather than an enforced block.

Typically a few days to four weeks. Submitting your sitemap and requesting indexing in Google Search Console usually speeds this up significantly.

Not directly, and not quickly. To remove pages that are already in Google’s index, use the Removals tool in Google Search Console, and allow the noindex tag to do its work over the following weeks.

Yes, but not on its own. Password protect the staging site at server level as well, or use the staging environment your host provides. The discourage setting alone leaves the site publicly viewable.

Yes. Yoast, Rank Math and All in One SEO all let you set a page to noindex individually – usually under an “Advanced” tab in the SEO box on that page. That’s the right tool for thank-you pages, internal documents and duplicate content, and it’s far safer than the site-wide switch.

That line tells search engines not to crawl any part of your site. On a WordPress site it’s almost always because the discourage setting is switched on. Untick it and the line will disappear.

No. Paid ads run independently of organic indexing, so your ads will still show. It’s not unusual to find a business paying for Ads traffic while their organic listings are switched off entirely – an expensive way to run a website.

Not sure whether your website is working as hard as it should be? Get in touch for a website health check and we’ll tell you exactly what’s helping and what’s holding you back.