Distinguish crawl rules from indexing controls before changing a live website.
Define what you want to achieve
Before editing a rule, decide whether the goal is to limit crawling, prevent indexing or protect private content. These are different requirements. Write down the exact pages involved and who should be able to access them. A configuration copied from a staging website can unintentionally affect important live pages.
Understand the difference
Robots.txt manages crawler access and is not a reliable way to keep a URL out of search results. A blocked URL can still be known through links. If using a noindex directive, the crawler needs access to read it. Private information requires proper access controls rather than public crawler instructions.
Review rules with a developer
Check the active robots file, page-level directives and server responses together. Test representative service pages and articles after any configuration change. Avoid broad rules until their scope is clear, and keep a copy of the previous configuration so an accidental change can be reversed promptly.
Add the check to every launch
During a release, inspect the homepage, a service page and a recent article. Document which pages should be indexable and which should remain restricted. Include this review when changing hosting, deploying a redesign or installing an SEO plugin. Monitoring should continue after launch because indexing reports do not update instantly.
Need help applying this to your website? Explore our SEO Services and Website Maintenance, or discuss your requirements with our team.
Reference: Google documentation: Crawling & Indexing.