WordPress Prevent Site From Being Crawled

To prevent your site from being indexed or crawled by search engines do the following:

  • At the WordPress level run this: Go to <site>/wp-admin/options-privacy.php.
  • From the WordPress Dashboard: Settings > Reading > Check: Search Engine Visibility (Discourage search engines from indexing this site).
  • Add a Robots.txt file to the root of your site containing the following 2 lines:
    • User-Agent: *
    • Disallow: /