« Musings between flights | Main | Style »

The poverty of surveys

Ever filled out an employee satisfaction survey?  I've lost count of the time I have wasted on these over the years, both in completion and interpretation.  The questions asked contain a hypothesis as to your motivation. Does your manager consult you on a regular basis? to which a response ranging from all the time, to not at all is expected.  My response was always along the lines of it depends. However the subtleties of context were not allowed and I was told to answer it on average or overall, to my mind such an instruction invalidates the input.  Interpretation was worse: Your division shows a 5% decrease in employees who are satisfied with their overall terms and conditions, but 10% more feel that they are treated fairly (that is a real case).  What is one meant to make of that?  Overall too many assumptions are built into the questions and the results provide numbers without context and this the lack meaning.  Fortunately there are now alternatives, but more of that later.

At lease the satisfaction survey is, in general well motivated.  Employers do want to know what is happening amongst their employees.  However there is another, and scurrilous use of the survey instrument, namely the supposedly objective facts disguising a sales pitch.  A good example of this can be found here in Melcrum's survey of employee engagement, linked to the publication of their  Practitioner's Guide to Employee Engagement .

I have extracted the following quote from the survey:  The survey revealed that 86% of organizations with employee engagement on the agenda are using at least one of the four most widely used techniques – action teams, Appreciative Inquiry, message maps and storytelling.  Now I would like to know on what basis that claim is made, it does not appear that this conclusion arose from unprompted responses to the survey.  It does look as though respondents were asked about those four items specifically, and surprise, surprise those just happen to be four chapters of the guide.

Story Telling it turns out is in use on 49% of the survey subjects.  Now the naive and over enthusiastic reader might take that as an endorsement of the subject, but is it really?  The survey is based on 1,625 responses from professionals working primarily in communication or HR.  So if anything it is surprising the response on story telling was not a lot higher.  I can't imagine anyone in communication not answering yes to their use of story telling so if the split is 50-50 between communications and HR then its bad news for the technique in HR departments.  In any event only 27% of the respondents even have an employee engagement programme in the first place.

Mind you, given that story-telling and appreciative inquiry are techniques designed to achieve management objectives, not to allow the authentic voice of the employees to to be heard I can sort of believe that the figures may be true.  Having executives tell their employees a story has always seen to be an very impoverished use of narrative, better, and more honestly described as propaganda.  Requiring your employees to tell positive stories (Appreciative Inquiry), refusing them the right to tell negative stories represents another Orwellian intervention.

Generally these type of handbooks are a process of dumbing down anyway.  One of the authors, a good friend of mine Tony Quinlan, on his blog bewails the process of simplification that he was required to go through.  It looks like he took the position that it was better to write the article with qualifications, rather than let a platitudinous charlatan take up the pen, and I happily support him in that.  He makes a good reference to the Mullah Nasrudin story of the Falcon to illustrate his point.

The ultimate condemnation here is the price.  Melcrum want £225 for four articles and a survey.  Now most of the material in those articles (if not all) can be obtained from the wikipedia and various articles available on the internet.  If you want to go beyond that then four books covering the four subjects would cost you less than £100.  This is a con trick.  Regrettably most of the publishers are going down this route.  Ark (who should know better) are allowing Uncle Jerry to publish the contents of an open, free access list serve at a similar price!

There really are better ways to do this.  Narrative work in organisations is about understanding and extracting meaning from  naturally told anecdotes (the ones they want to tell, not the positive ones demanded by the appreciate inquiry process) that allows people to make decisions.  Numbers with context come from narrative based research, and that enables action.  Complexity techniques such as social network stimulation (we teach that on the Cardiff course along with narrative) allow employees to engage with colleagues across silos and outside the formal boundaries of the organisation.

One day, sanity may break through ....

Comments (3)

Hi Dave,

Not long ago I wrote a post related to staff satisfaction surveys

There are a few things wrong or old with this approach:

- It seems an uninteresting thing to take part in, so this will effect take-up
- If I do take part, how do I know they care, what’s the follow-up process
- It is structured (not free form), I’m responding to an agenda
- The responses are not shared with others (assumed management is the oracle that can answer your questions)

http://libraryclips.blogsome.com/2008/01/15/staff-satisfaction-using-km-20-networks/

Jon Husband:

It's quite astonishing when you get into it and explore almost all surveys' superficial construction. Coupled with the range of misinterpretations and arbitrary choices any given employee can make, multiplied by many many employees, and how flimsily not-useful the results almost always are.

Today one can legitimately consider using methods such as the aggregation and summarization via text analytics of actual content from social computing, crowdsourcing of anecdotes and opinions, narrative capture per the Cognitive Edge approach.

I am willing to bet that the putcomes of any of these methods, treated with contextual and designed-constraint filters and / or some ordered and relatively rigorous statistical interpretation (to cluster and then extract sense) will yield much more useful portraits and sense about what is actually going n and what people think and feel.

But the big consulting firms know very well how to sell survey making, survey implementation and survey interpretation, and quite often have established and compliant buying channels.

I'm really just parroting what you have said, but the combination of relative uselessness and expense of surveys has irritated me for quite a few years now and so I thought I'd vent a bit.

Hi all

satisfaction is a mindset or at least a state of mind (english is not my native language, so I hardly make the difference)

Satisfaction surveys report symptoms, the average (!!!) of which is supposed to report a mindset: I can't accept this kind of mixing apples and pears.
Satisfaction surveys measure something else - I do not know what - but satisfaction.
Same than you, I gradually rised doubts about my filling endless surveys. Since it means nothing, results are not actionable and let's be frank, I always heard "nothing has been done with our inputs". Luckily, may be.

Kind regards
Rene

Post a comment




Remember Me?

(you may use HTML tags for style)