« Sometimes it just works …. | Main | Hoist on my own petard »

The consequences of measurement

Goodhart a British economist is widely held to have formulated the equivalent of Heisenberg’s Uncertainty Principal for economics. It states as follows

Any statistical relationship will break down when used for policy purposes

A simpler formulation from a US academic translates this as:

If a measure becomes a target, then it ceases to be a measure

Now if I had my way I would have this engraved on tablets of stone and placed in the entrance hall of every Government department in the world. If I just look at the press this week we have had (i) stories about the Ambulance service in Britain misusing the classification system to achieve targets and worst still (ii) a UK Government Think Tank saying that although all key targets for the Health Service have been met, the overall service has deteriorated. The problem is compounded by a view of measurement that is stuck in the hypotheses-empirical tradition of evidence. An approach that has value in some cases, but little credibility in conditions of uncertainty and complexity. My next Frontiers article will address this, looking at ways to achieve un-gameable measurement systems that represent complex environments without assuming linear causality.

In a very real sense you get what you measure and you live to regret it. In the case of government we are the ones to suffer. Of course it could get worse. When Six-Stigma starts to loose its shine then Governments will adopt it as industrial best practice and after that the bottom 10% of all patients will be culled for the sake of the health system as a whole.

TrackBack

Listed below are links to weblogs that reference The consequences of measurement:

» If it doesn't work, do more of it with greater vigour from Cognitive Edge
I see that Dilbert has picked up on Six Sigma. Or, to use my deliberate slip of the toungue (to which I confessed in a response to a recent comment): Six Stigma.... [Read More]

Comments (14)

paul Anderson:

David,
Greetings from mackay, Queensland. We met at your Dec/05 session. I involved with the Dalrymple Bay Coal Terminal in Queensland
I eagerly await your Fronteirs article. I'm on a steering group trying to implement a 'whole of coal chain' approach to improving cola flow down the Goonyella Coal Chain system here in Qld. The 'gaming' is amazing to watch and experience as terminal users jockey to achieve commercial aims.
Your reference to "ways to achieve un-gameable measurement systems that represent complex environments without assuming linear causality" is right on the button for me.
Cheers
Paul Anderson
As well the Dalrymple Bay terminal, previously run on or around complex adaptive systems thinking and principles, is moving towards Six sigma in the pursuit of greater throughput.
Any chnace of an early release of your article.
Your

David Williams:

OK, if I don't know then someone else won't either, so I'll ask the question - is Frontier a Journal? If so where can I find it? Thanks!

Dave Snowden [TypeKey Profile Page]:

Sorry David - Frontiers is a piece I write for each edition of E:CO. You will find past versions on the resources section of the web site. The next one is due to the publisher shortly and will then be posted on the web site once its formated

Britta Mohr:

Engraving your message on tablets of stone does unfortunatley not fit into our IT infrastructure, but I am thinking about putting it somewhere into our Balanced Scorecard IT solution as a constant reminder to people when they are creating Key Performance Indicators.

;-) Britta

Hi -

"...un-gameable measurement systems that represent complex environments without assuming linear causality..."

They already exisit. They are called markets.

-j
http://www.pmcluster.com/

Dave Snowden [TypeKey Profile Page]:

If that was the case John you would not need the SEC or other regulatory bodies.
And if you want to see the consequences of an open market in public service provision many of us can point you to the downsides .....

Great link Dave,

Those Goodhart quotes are as good a statement of that issue as I've seen. I tend to refer to it as our "Catch-22" in knowledge management.

"You get what you measure" is good too.
Did I ever refer you to Dr James Willis for some good UK healthcare perspectives on this - from an experienced GP.

Regards
Ian

Lauchlan Mackinnon:

David,

I just thought I'd mention in passing that I addressed Goodhart's law and the general princile underlying it in detail in my Ph.D thesis - see chapter X and Appendices B and C in the pdf downloadable from http://eprint.uq.edu.au/archive/00004455/ if you are interested.

You might also like to look into, if you haven't already, the Lucas Critique, Merton's Nobel Prize address in economics, and the "reflexive prediction" literature around Morgenstern and Grunberg and Modigliani. These contributions were also all reviewed in my thesis.

I think from a KM perspective you might also find something of interest in my chapters III-V on how ideas enter into the process of social construction of economic / business / political systems.

Dave Snowden [TypeKey Profile Page]:

Thanks for the thesis Lauchlan. You broke one problem for me on the book in the opening pages by referencing Einstein's "riding the waves of light" which I had forgotten as its a great example of metaphor.

I only skimmed it I am afraid (constraints of time) but I would be interested to know if in rejecting critical realism (with which I agree) and identifying the role of social construction, you then flip over into social constructivism? Your other note referencing post modernism was ambiguous on this. You have some interesting connections between what constitutes ontological status in economic ideas and social construction of those ideas that also act as indicators but not proof!

On the questions of economic agents you pick up on Mill, but Williams attack on Mill (Philosophy not Economics) is not mentioned and there are some links here to the consequences of social atomism (Weissman) that it would be interesting to see as I think it would undermine some of the subsequent development of the concept of "rationalism" in economic theory.

In other words I wonder if you are wedded to the concept of "agent" in the sense of Mill and successors and are using social construction as a means to overcome the phenomenological weaknesses of that approach?

Ideational reflexivity is I think better explained by complexity theory, which I think also gives (with emergence, co-evolution and Irreversibility) a better account of social construction that we get from social science orthodoxy. The treatment of Brian Arthur by the Economic orthodoxy is an interesting example of the ability of a idea to sustain itself rather like a white blood cell defense.
It would be interesting if you do move onto a book (or a paper) to see how you would treat complexity, and also the pattern basis of human decision making. Both are recent in source but there is a body of literature (and even a journal or two) in economics which is exploring this field.

Lauchlan Mackinnon:

Hi Dave,

There's quite a bit to think about there, but a few quick points:

I don't feel that I reject Critical Realism per se (in fact, I like the ontological approach and I think it makes some great points), I just think that Bhaskar's version of it is excessively abstract and at a philosophical/methodological level has some major flaws (some of which Margaret Archer addresses in her work, for example the temporal sequencing of events where structure and agency interact on each other and some others of which I touch on in my Ph.D).

In my thesis, I argue that the flaws in Bhaskar's approach are precisely what preclude it from being a useful framework for understanding how ideas enter into the process of social construction of economic systems.

With regards to Mill, my thesis traces the development of the economic agent ('homo oeconomicus') from Hobbes through to Arrow and Debreu (General Equilibrium theory) in the 1950s, to argue that economists conception of the economic agent is a social construction. While this point may not in itself be radical, I do not believe anyone has traced through the development so explicitly. J. S. Mill appears as an important figure in that development.

Economists are almost universally committed to what they call 'methodological individualism' with notable exceptions found only in areas such as (old) institutional economics, Post Keynesian Economics, Marxist economics, and Evolutionary economics.

That is, they take the assumption of the 'rational' maximising agent as the theoretical starting point and major assumption. Although I don't really spell it out as much as I should, my thesis is in part an argument against that conception by developing an account where the agent is socialised into and participates in reproducing and evolving social structures, and that the approaches developed by Bhaskar/Bourdieu/Berger and Luckmann provide an appropriate framework in which to conceptualise this relationship.

For similar work along those lines in economics appealing to a Critical Realist / ontological framework for support, you might be interested in 'Reconstructing Economic Theory' by Allen Oakley and 'The Theory of the Individual in Economics' by John B. Davis .

Thanks for the suggestions to consider how a complexity approach would consider the same issues. At the University of Queensladn (where I did my Ph.D) we had a number of people interested in and active in the complexity approach, such as Jason Potts (http://www.uq.edu.au/economics/index.html?page=15895&pid=15834) and Professor John Foster (http://www.uq.edu.au/economics/index.html?page=15870), so I have had some exposure to complexity ideas. I would be interested to hear how you see the complexity approach articulating the relationship between structure and agency. My biggest philosophical issue with the complexity approach within economics is that by and large it is still a mathematical metaphor (although a better and more realistic one) and as such still tries to 'fit people into equations' - a task I view with some suspicion!

Dave Snowden [TypeKey Profile Page]:

I fully agree that the concept of a rationalising agent has to be challenged - and "evolutionary economics" which draws to varying degrees on complexity theory is starting to do that. However I don't think the shift a socially constructed agent is radical enough. It still makes a atomistic assumption about identity residing in an individual rather than in a system. What I call computational complexity I agree is too much of a mathematical construct. Other approaches such as Stacy (Participative complexity) and my own (contextual complexity) move beyond that into social complexity which includes links to human cognition and decision making. This approach basically says that you cannot separate strucutre from agency, the two form part of a co-evolutionary network. I think incidentally that Williams devasting attack on Utilitarians in a classic "Morality" is a prequal to to complexity attack on notions of atomistic identity. http://www.cognitive-edge.com/articledetails.php?articleid=40 has some basic thinking on this

Lauchlan Mackinnon:

"my own (contextual complexity) move beyond that into social complexity which includes links to human cognition and decision making. *This approach basically says that you cannot separate strucutre from agency*, the two form part of a co-evolutionary network"

This is the part I'm most interested in. :)

How do you conceptualise that? This is where I use frameworks such as critical realism, Berger and Luckmann's social constructionism, Bourdieu's habitus/field, or Giddens's structuration. I am interested in how you approach the same tasks.

I appreciate if you haven't got time to get into it now - it is probably a fairly lengthy description.

Regards,

Lauchlan Mackinnon

Dave Snowden [TypeKey Profile Page]:

Lengthy - and some of it is being written for the book as we speak
Some of the articles on the web site (that referenced) pllus the more substantial New Dynamics of Strategy are part of the way there. I am writing a new Frontiers piece at the moment which will address aspects of research and measurement per se. But a lot of it is not fully articulated yet - but the more questions the better it will be and your thesis was a help as it provided a very precise and welcome summary of the field

Lauchlan Mackinnon:

Hi,

Glad I could help! :)

I'd be interested to ead your books/papers on the topic when they come out.

Lauchlan Mackinnon

Post a comment




Remember Me?

(you may use HTML tags for style)