Google’s robots changes, the web & the law

Google announced a few days ago that they would change the way rel="nofollow" works for them. They would start treating nofollow as a “hint” instead of a “directive”. This means they go from “oh you don’t want us to go into that room? ok” to “oh you don’t want us to go into that room? We’ll see about that”. Basically, they went from friendly neighbor to annoying parent real quick.

Now rel="nofollow" is something you’d attach to links on your site. So I could nofollow one link and not nofollow another. There is also a meta robots directive, used like this:

<meta name="robots" content="noindex,nofollow"/>

This would historically direct Google to not show that page in its index, and not follow its links. Now Gary Ilyes tweeted this last night:

1. There's no meta robots ugc and sponsored, it won't do anything if you add that.
2. Meta robots nofollow is a hint now, like rel-nofollow.
3. I'll update the docs tonight to say this explicitly.

— Gary 鯨理／경리 Illyes (@methode) September 12, 2019

When Gary says “meta robots nofollow is a hint now”, I become slightly nervous. Because if nofollow in a meta robots element is a hint, what is noindex? Do they now want to treat that as a hint? or as a directive? Turns out, noindex remains a directive:

If you use say noindex in a meta tag, we won't index the page. It's a command we continue to honor.

If you say nofollow in a meta tag, that's going to make us see links on the page as nofollow links — and we'll treat the application of that to the *links* as a hint.

— Danny Sullivan (@dannysullivan) September 11, 2019

But, even if Google now says “noindex will remain a directive”, won’t that lead to years and years of discussion? In my experience, even many experienced SEOs don’t always understand the difference between directives and hints, and think they’ve excluded something when they haven’t. This change will only make this worse.

Google unilaterally makes changes

My biggest gripe with this is that Google is making these changes unilaterally. Bing, Yandex, Baidu: all support rel="nofollow" and other search engines probably do too. The same is true for meta robots nofollow. I don’t think it’s a good idea when Google decides on its own that it changes the “laws” of the web.

Google were the ones to introduce rel="nofollow", which gives them some rights to change that “standard”. However, meta robots nofollow has been around since 1996. In fact, this part of the meta robots page made me chuckle:

robots can ignore your <META> tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.

Taken from the Robots pages

Apparently, I should consider Googlebot malware from now on 😉

Real world implications

Let’s look at real world implications: a link in a comment on a WordPress site used to have rel="nofollow" added to it automatically. We’ll now have to change rel="nofollow" to rel="nofollow ugc". We can’t take out the nofollow, because other search engines don’t support the ugc part, but Google, with its market domination, will urge us to make that ugc change.

Now Google’s first reply to this will be “you don’t have to change anything if you don’t want to, we even said that in our post”. And they did:

There’s absolutely no need to change any nofollow links that you already have.

Google’s Danny Sullivan in their announcement blog post on the nofollow changes

I read this and chuckled. Obviously Google needs to read up on the murder of Thomas Becket. Because of Google’s market dominance, people will do anything to get into their favor. They can make changes like this and the web will follow. The real question here is: shouldn’t we have legislation that prevents them from making these changes unilaterally?

In fact I’d say it’s time to go one step further: the web needs to have true standards for this. Standards that are preferably turned into law by the European Union, the US and China. But I dream too much perhaps. They at least need to be standards. Standards that all non-malware crawlers, including Google, will adhere too.