Are your website’s hidden pages appearing in Google?

Google’s ending support for robots.txt Noindex — how does it affect your site?

Are your website’s hidden pages appearing in Google?

Web developers were long able to put a list of any pages they didn’t want appearing in search results into the website’s robots.txt file. That made it easy to keep all of your page access control in one place. But that’s changing very soon, so you’ll need to take action if you don’t want your hidden pages suddenly appearing in Google searches.

The problem with the robots.txt file is that it never became an official standard. Nobody is in control of it, so there’s no one set of rules as to how it’s used. Also, its security wasn’t so great. They’re publicly available files on your web server that give away the entire list of pages you don’t want people to see. That’s obviously not a great way to protect your website content.

“In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we’re retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019."

That was Google’s announcement and while it’s generally a good thing, it might make some previously hidden areas of your website suddenly start to appear in Google’s public search results.

So what do you need to do?

Fortunately, there are plenty of more secure, industry-standard ways to protect parts of your website. The use of on-page meta tags is better for individual pages that you want hidden but aren’t so secure that you’d need a password. Password protection is available to limit pages to specific users or groups. And you can remove URLs from Google directly through Search Console. Ask your web developer what’s best for your site.

The robots.txt file still has plenty of uses, though. The Disallow command remains the best way to forbid search engines from indexing entire sites or large sections of your site. It’s also the place to stop the indexing of certain types of file on your web server. And last but not least it’s best practice to list your site map there, to give search providers a clear signpost to the list of pages on your site.

3. Write a daily list

I’ve found that working straight from my Trello list tends to end with me getting distracted by my other tasks. So every morning I look at my task list and write down (using pen and paper like some sort of caveman) the three vital tasks that I MUST do that day.

This has been far more effective than I expected.

By physically writing down your tasks, you start your brain engaging with them, and you’re left with a distraction-free list of work to do. Only once those vital things have been done do you go back to the master list. Or take a nap. Whatever takes your fancy.

4. Kill the distractions

We can’t multitask.

Really, we can’t. Our brains just won’t work that way. And the nearest we can get — rapidly switching between several tasks — leaves none of them getting done well. In fact, studies have shown that after switching tasks it can take our brains up to 20 minutes to settle in to proper work again.

And we can only concentrate for about 25 minutes. Seriously, our brains are rubbish at focussing on lots of things at once AND at focussing on one thing for a long time.

So, we work with what we’ve got, using this variant of the Pomodoro technique:

NOTE: The original Pomodoro technique says to do four 25 minute sessions then take a long break, but I’ve found it easier to concentrate when I take my long breaks between tasks. Your mileage may vary.

Further reading

There are some more great tips for motivating yourself at work in this article from Toptal:

Triggering Productive Behavior: Motivation Tips for Work >