One of the skills that we need present within the Network is questionnaire design, for use with the SenseMaker™ software. It is clearly critical that there is credibility in the way SMIs are tagged on their entry in the system. If there is any question about that credibility, then it throws the results and any insights gained into doubt.
There are many schools of thought and ideological positions about best practice in the design of questionnaires. We have experimented with different ways of presenting filters in the past. For example, we’ve used sliding bars to indicate the degree of presence of AVTs but there are concerns about whether or not it can be assured that the indexer has actually moved the sliding bar from its resting position when first presented. We think this can be countered by using new technologies so the slider does not appear until touched by the indexer and will be introducing that shortly (though I don't think it will resolve the argument!).
We’ve also used radio buttons for filters. These have the advantage of being presented as ‘blank’ to the indexer, so when an entry is made we can be assured that the indexer consciously entered the result into the software. However there are only so many radio buttons that can be presented on a webpage and it is necessary to add a small variable around the results to display properly. While this doesn’t change the results in any significant ways (the patterns are more important than the precise statistics) it does result in questions about the results.
The best way to design the multi-choice questions in a SenseMaker™ questionnaire though is probably under-considered. This was pointed out to me recently when I met with one of the talented analysts in the Psychological Assessment and Research Branch of the MOE in Singapore. As an example, he said that multi-choice questions should never have an odd number of answers, because it increases the prevalence of people choosing the ‘middle’ answer from those presented.
Is this a valid point? I’m not sure but I’m certainly prepared to entertain it. If you have evidence of this, and of any other recommended practices in questionnaire design, I would certainly appreciate it if you could pass them along.
Comments (1)
Hi Steve
There is a vast amount of academic literature on QS design and how an individuals cognitive bias impacts on how they approach the QS.
This journal in particular carries papers from practicing researchers on QS issues
http://www.ijmr.com/
As we have discussed previously, there are several methodological considerations to take into account when indexing data in Sensemaker. My feeling at the minute is that when using Sensemaker to analyse data, practioners need to have a VERY good understanding of the processes that got the data there in the first place.
Posted by Stuart | September 29, 2007 8:52 AM
Posted on September 29, 2007 08:52