Are your website’s hidden pages appearing in Google?
Web developers were long able to put a list of any pages they didn’t want appearing in search results into the website’s robots.txt file. That made it easy to keep all of your page access control in one place. But that’s changing very soon, so you’ll need to take action if you don’t want your hidden pages suddenly appearing in Google searches.
The problem with the robots.txt file is that it never became an official standard. Nobody is in control of it, so there’s no one set of rules as to how it’s used. Also, its security wasn’t so great. They’re publicly available files on your web server that give away the entire list of pages you don’t want people to see. That’s obviously not a great way to protect your website content.
“In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we’re retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019."
That was Google’s announcement and while it’s generally a good thing, it might make some previously hidden areas of your website suddenly start to appear in Google’s public search results.
So what do you need to do?
Fortunately, there are plenty of more secure, industry-standard ways to protect parts of your website. The use of on-page meta tags is better for individual pages that you want hidden but aren’t so secure that you’d need a password. Password protection is available to limit pages to specific users or groups. And you can remove URLs from Google directly through Search Console. Ask your web developer what’s best for your site.
The robots.txt file still has plenty of uses, though. The Disallow command remains the best way to forbid search engines from indexing entire sites or large sections of your site. It’s also the place to stop the indexing of certain types of file on your web server. And last but not least it’s best practice to list your site map there, to give search providers a clear signpost to the list of pages on your site.
