One way to think about AI's unwelcome intrusion into our lives can be summed up with Godhardt's Law: "When a measure becomes a target, it ceases to be a good measure": 1/

If you'd like an essay-formatted version of this thread to read or share, here's a link to it on , my surveillance-free, ad-free, tracker-free blog: 2/

Goodhart's Law is a harsh mistress. It's *incredibly* exciting to discover a new way of measuring aspects of a complex system in a way that lets you understand (and thus control) it. 3/

In 1998, Sergey Brin and Larry Page realized that all the links created by everyone who'd ever made a webpage represented a kind of latent map of the value and authority of every website. 4/

We could infer that pages that had more links pointing to them were considered more noteworthy than pages that had fewer inbound links. 5/

Moreover, we could treat those heavily linked-to pages as *authoritative* and infer that when they linked to *another* page, it, too, was likely to be important. 6/

This insight, called "PageRank," was behind Google's stunning entry into the search market, which was easily one of the most exciting technological developments of the decade. 7/

The entire web just *snapped* into place as a useful system for retrieving information that had been created by a vast, uncoordinated army of web-writers, hosted in a distributed system without any central controls. 8/

Then came the revenge of Goodhart's Law. Before Google became the dominant mechanism for locating webpages, the only reason for anyone to link to a given page or site was because there was something there they thought you should see. 9/

Google aggregated all those "I think you should see this" signals and turned them into a map of the web's relevance and authority. 10/

But making a link to a webpage is easy. Once there was *another* reason to make a link between two web-pages - to garner traffic, which could be converted into money and/or influence - then bad actors made a *lot* of spurious links between websites. 11/

They created linkfarms, they spammed blog comments, they hacked websites for the sole purpose of adding a bunch of human-invisible, Google-scraper-readable links to pages. 12/

The metric ("how many links are there to this page?") became a target ("make links to this page") and ceased to be a useful metric. Goodhart's Law is still a plague on Google search quality. 13/

"Reputation abuse" is a webcrime committed by venerable sites like Forbes, Fortune and Better Homes and Gardens, who abuse the authority imparted by tons of inbound links accumulated over decades by creating spammy, fake product-review sites stuffed with affiliate links. 14/

Google ranks these more highly than real, rigorous review sites because of all that accumulated googlejuice: Goodhart's Law is 50 years old, but policymakers are woefully ignorant of it and continue to operate as though it doesn't apply to them. 15/

This is especially pronounced when policymakers are determined to Do Something about a public service that has been starved of funding kicked around as a political football to the point where it has degraded and started to outrage the public. 16/

When this happens, policymakers are apt to blame public servants - rather than themselves - for this degradation, and then set out to Bring Accountability to those public employees. 17/

The NHS did this with ambulance response times, which are very bad, and that fact is, in turn, *very* bad. The reason ambulance response times suck isn't hard to winkle out: there's not enough money being spent on ambulances, drivers, and medics. 18/

But that's not a politically popular conclusion, especially in the UK, which has been under brutal and worsening austerity since the Blair years. 19/

(Don't worry, eventually they'll do enough austerity and things will really turn around, because, as the old saying goes, "Good policymaking consists of doing the same thing over and over and expecting a different outcome.") 20/

Instead of blaming inadequate funding for poor ambulance response times, politicians blamed "inefficiency," driven by a poor motivation. 21/

So they established a metric: ambulances must arrive within a certain number of minutes (and they set a consequence: massive cuts to any ambulance service that didn't meet the metric). 22/

Now, "an ambulance where it's needed within a set amount of time" may *sound* like a straightforward metric, and it was - retrospectively. 23/

As in, we could tell that the ambulance service was in trouble because ambulances were taking half an hour or more to arrive. But *prospectively*, after that metric became a target, it immediately ceased to be a good metric. 24/

Ambulance services, faced with the impossible task of improving response times without spending money, started to dispatch ambulance *motorbikes* that couldn't carry 95% of the stuff needed to respond to a medical emergency, and had no way to get patients back to hospitals. 25/

My favorite example of this is the AI-powered Roomba that was programmed to find an efficient path that minimized collisions with furniture, as measured by a forward-facing sensor that sent a signal whenever the Roomba bumped into anything. 28/