Little has been written on how to develop a process based on the logic and method of SenseMaker. I remember being intrigued and increasingly excited by how it might possible be useful. But the details of how to do this, even after attending the course, were murky. Diving in over the past two years has been a fun and insightful but not terribly efficient way of learning how it works and what is needed to make it work. So I’ve been developing a short ‘here’s what you need to think about’ set of headlines when I discuss and design with organisations.

So far, my experience is that a good design boils down to three elements that need careful thought. And in fact, thinking this through might be a good way to assess if Sensemaker makes sense in the first place.

Challenge 1. Open and evaluative question framework.
I’ve likened developing a signification framework to sumi-e that asks of the artist to strip the object down to a few essential brushstrokes ‘to capture its soul’. Getting to the minimum set of question options means letting go of one’s survey mentality and linear correlation thinking. Two other issues are important:
•Who’s in the room? All my applications of SenseMaker are meant to be useful to a range of organisations. So far, limited time and money means I haven’t been able to design a framework collectively with the intended audience. So this means solid pre-testing and then fieldtesting with them to get feedback.
•A really tricky issue when used for evaluation, as compared to more open-ended research, is the tendency to frame questions too blatantly as evaluative. For example, in eastern Africa, one organisation I’m working with wanted to ask ‘what made you feel hopeful or discouraged about X programme’. Quite counter to the abductive reasoning on which SenseMaker is based. We seem to have less gameable questions now that still allow us to come to evaluative insights.

Challenge 2. Stories at scale
Without lots stories, SenseMaker is a waste of time and money. So getting good story processes to get to enough scale is crucial. The GlobalGiving system works through volunteer scribes who get a minimal reward per story. But for another organization seeking narratives from its partner organizations, the relationships with those organizations become part of the puzzle. You rely on them to find stories and that means they need to be motivated. Or in our water sector application, you need to think of how to motivate water professionals at district and national level to share their experiences on a regular basis. We’ve got a range of ideas that we are and will be trying, ranging from straightforward sub-contracting to a research firm to students and junior professionals.

Most SenseMaker applications have been for one-off research projects. I’m applying it for ongoing impact-oriented monitoring. So whatever system emerges, it needs to be resilient and efficient and sustained.

Challenge 3. The ‘right people and ‘right patterns’ to make it useful
Any evaluative process, if not used by someone, is pretty useless. Utility is part of the evaluation standards for a reason. So thinking through the learning processes, once stories are collected is crucial. And I’m convinced this needs to start early on. For one organisation working on smallholder agricultural development, story sources and story users were clearly two sides of the same coin. A discussion we had even before we started developing the question set.

We discussed:
•Who will be asked to look at patterns and stories? I.e. where do you want to stimulate the kind of reflection that story pattern analysis can provoke?
•How do you build in regular reflection that you can feed by showing how story patterns shift? Is it an annual staff meeting, a bi-annual workshop of companies and farmers, a regular policy forum, etc?
•Even a small set of 12 or 13 questions generates a wide range of possibly interesting patterns. As there is never enough time to look at them all, which patterns are the ‘best ones’?

Each process has its own sticky points. In the GlobalGiving work, the third challenge needs more work. How do we make GlobalGiving’s partner organisations curious about what the stories reveal? In the water work, we really have to think hard to get to a feasible and affordable system to ensure stories come in regularly. In the girl’s empowerment application, the client wanted more directly evaluative questions. How do we meet that need and build in space for the open questions that can surprise and trigger innovations?

Being involved in so many applications is enriching because of the immediate comparative advantage it offers. But I’m in danger of only seeing nails, now that I am wielding the SenseMaker hammer. So I’ll turn tomorrow to some editing work of 8 case studies on ‘normal’ evaluation practice. And after that a blogpost on a much larger question than that of design – does SenseMaker actually help in complex situations?

< Prev

Evaluating SenseMaker

As I mentioned, 2011 is my year of action research on SenseMaker. I’ve got six ...

Category:

Further Posts

Next >

Max Boisot 1943-2011

It has taken me some days to be able to write this, and I ...

Category:

Further Posts