Qualitative Quantitative Evaluation (QQE)
QQE stands for "Qualitative Quantitative Evaluation" because it combines qualitative (comparing) and quantitative (counting) methods of evaluating the results.
The purpose of QQE is to link what you have discovered to things that decision makers consider important and pay attention to. This is done by creating a grid relating each "object" previously discovered to important functions or processes of the organization. By considering patterns that form in the grid, you will derive a portfolio of projects or interventions that, according to your descriptive self-awareness, will have the greatest utility.
To populate the columns in your QQE grid, generate a short list of "things that matter" to the organization - either things decision makers value or things that matter in a more abstract sense. These will be typically "core processes" or "essential functions". There are two ways to do this.
You can gather material separately from different decision makers, then bring the results together and present them back to the decision makers as a group. Encourage dissent and initiate discussion around the list to refine it. Then ask the decision makers to score each agreed-upon item based on its importance to the organization. Use the same scale you used for the knowledge objects.
Or, you can derive character archetypes from anecdotes related to the issue at hand. Gather a group of people and go through the same process as described for decision makers above, except ask the people in the group to represent the viewpoints of each archetypal character as if it were one of the decision makers. This can also be done with themes or values as well as character archetypes - in that case each "decision maker" is someone who cares primarily about one theme or value.
One possibility is to do both of these processes - with both real and archetypal decision makers - and then either merge or juxtapose the results, for a richer result. At the tops of each column of your grid you should now have the name of a core process or essential function and an "importance" score for it (possibly as a color code).
The table below describes the various stages used in the method together with commentary where appropriate.
| Task | Comments | |
|---|---|---|
On the rows of your grid place the objects which arose from the disclosure points and perspective-question answers. These will also have a "score" based on the evaluation criterion for the object, such as its vulnerability to loss (knowledge) or controllability (dynamics). Before you start calculating your grid, stop and have people agree on a set of criteria that will determine what patterns are strong enough to take action on. Would three medium-to-high values represent a pattern, or would four be required? Make sure this decision is made before you start, because otherwise people may be tempted to change the rules to fit their prior expectations or agendas. | ||
Now combine the values in the grid. If you used a numerical scale, you can multiply the numbers at each intersection to achieve a relative combined "score". For example, say you are looking at a "cell" of the grid in which the column (essential function) "ship on time" has an importance value of 5 and the row (dynamics object) "relationships with suppliers" has a controllability value of 3. The combined value would then be 15. If you used a qualitative scale, you can create simple "rules" for combination, such as "the combination is the highest of the two scores", thus L+M=M and L+H=H, or "the combination is the rounded average of the two scores converted to numbers (123)", thus L+M=(1+2)/2=1.5=M and L+H=(1+3)/2=2=M. Basically whatever everyone accepts as meaningful is meaningful. | ||
In any case, color-code the cells in the grid based on the range of numbers or qualitative levels they cover. As you look at the grid, patterns will emerge naturally. And they must be allowed to emerge naturally; any pattern that meets the criteria you set up earlier is allowed, which means no one can artificially create or deny one. Vertical bar patterns are things that are easy to sell because they obviously impact an important function, but are hard to implement because they require intervention in several object areas. For example, there may be many ways to improve on-time shipping, but it may be difficult to tackle all of those ways at once. Horizontal bar patterns are hard to sell because they impact several functions but in minor ways, but are easy to implement because they only require intervention in one object area. For example, adding a supplier extranet might be a low-hanging fruit, but it will have only minor impacts on several essential functions. | ||
The patterns you choose from the grid as most deserving of action will form the basis for your portfolio of projects. | ||
Once you have identified a portfolio of projects which might prove fruitful to approach based on your QQE grid, you need to evaluate them to determine which you will actually carry out. To do this you can use two methods: a Cynefin framework (ideally contextualized, but generic will also work) and the perspective question used earlier in the mapping. Ideally you will use both methods and combine the results, but if you can only use one it is better to use the Cynefin framework method since it has not been used yet in the mapping process and thus provides another perspective.
To use a Cynefin framework to evaluate the projects in the portfolio, get a group of people to discuss them and distribute them across the Cynefin domains (and subdomains if possible). Use multiple domains or sub-domains if it seems useful or necessary to describe multiple aspects of each project. For example, perhaps a project to add a supplier extranet would have aspects in hidden-order space (technical details) as well as complex space (getting suppliers to use the extranet). Also ask the people to describe the projects using Cynefin dynamics patterns. Maybe the extranet project involves moving the supplier community from where it sits at the far edge of complex space closer to the hidden-order boundary by making more elements of relationships with suppliers transparent.
To use the perspective question to evaluate the projects in the portfolio, ask people to discuss and describe the elements of the acronym with reference to each project. For example, one attractor for the extranet project might be that suppliers would save time, and a barrier might be that employees would be reluctant to give up their old ways of conducting business with suppliers. Going back and rechecking each project against the same perspective question that started the mapping process is a good way to find out if you are still on the same track as far as your goals in doing the mapping.
In either of these types of evaluation, you have three goals:
- to make decisions about which projects are higher priority than others
- to check against entrainment leading the patterns you choose to pick out
- to identify common elements among projects (because finding those could reduce the difficulty of doing the number of projects you want to do and create common start points to multiple projects)
After people agree on which projects to carry out, have them discuss success measures and milestones, ideally by describing each project as a Cynefin dynamic with milestones represented by turning points in the dynamic. Also, have people talk about some of the risks involved in each project and generate some things to watch for in order to detect if things are not going as planned and the project has to be rethought or changed. Each project should also include a capture or review aspect in which lessons learned are accumulated.
CITATIONS
If you would like to leave a citation about this method you may do so here.
COMMENTS
If you would like to leave a comment about this method you may do so here.