I promised an attack blog for each of the twelve days of Christmas and received lot of suggested targets.   So there is obviously a market out there for taking down sacred cows, little tin gods and a paper tiger or two.  As I thought about the options while cooking the Christmas Dinner I mellowed a little.  Something to do with the good wine, or more likely the new Aga which maintains its heat over a long cooking cycle.  Its predecessor was over thirty years old and required careful nurturing to produce anything.   Now when I say mellow, I am not signifying any intention to slack off, or be nice for the sake of it.   Instead I started to focus on various sinful qualities and properties that I disliked at least or was deeply distressed by at worse.   I then realised that it might be better to hit those, and use the various offerings and approaches as illustrations of the different sins. So here goes …

1: Breaking Goodhart’s Law

Good hart was a leading British Economist and formulated the law during the Thatcher years.   It is better understood in the variation devised by Anthropologist Marilyn Strathern and I have my own variant, the meaning of which will become clearer after my first target example. 

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes  (Goodhart)

When a measure becomes a target, it ceases to be a good measure (Strathern)

Anything made explicit will sooner or later be gamed for survival purposes and that need will corrupt practice and people (Snowden)

This error has become increasingly common over the last couple of decades and can be illustrated by the Research Excellence Framework.  It was devised within the UK but there are equivalents in the US and elsewhere.   The various evils and stupidness of this system are legion, but have been well documented recently by Yiannis Gabriel for whom I have the deepest respect and have spent several fascinating hours in conversation.  So I am not going to summarise or extract from his post on this, go read it!  The various practices he documents there lead to Snowden’s variation on Goodhart:

The other example which is relevant here is Net Promotor Score or NPS.  The original Stanford Research showed that there was a strong correlation between answers to the ​Would you recommend our product/service and high performance.   The research also claimed that this was more accurate than most questionnaires and other measures, with some justification.  The problem happened when a reasonable measure became a target and all of a sudden managers were expected to shift a point on a Likert Scale with little idea of how.  Sooner or later the questionnaires stated to expand (thus contradicting the original research) seeking illustrations of practice.  Like many another questionnaire that created a market for analysis and yet more consultancy.

The reader can doubtless find other examples with ease.   At the moment we are working on an alternative to NPS which starts with the question but then focuses on non-explicit interpretation using SenseMaker®.  We have run the first successful project and critically it allows managers to ask What would I do to get my customers to tell more stories like these and fewer like those.   A much more pragmatic approach shifting the scale.   It also allows fast feedback loops and wider employee engagement.   The REF is more problematic but we are running trials on an alternative to the UK Student Satisfaction Survey which may give us some direction for the wider problem.

< Prev

Clearing decks for Xmas

The saga of Christmas for the last couple of decades has involved getting into town ...


Further Posts

Next >

Statistical tails wagging the dogs of truth

2: Confusing correlation with causation and the paucity of good data This seems to have ...


Further Posts