Qualitative research privileges the expert or experts in being able to analyze the content of data collected from interviews, focus groups, and other sources. This effort in itself is subject to cognitive bias as the researcher will see things in the content of such materials through the ‘lens of their expertise’. Another limitation is the fact that often the content is incomplete in terms of full context and it is likely to be ambiguous. Such content will then draw on the expert to ‘fill in the gaps’ with their interpretation which overlays meaning based on their expertise and judgement. The third limitation of typical approaches to qualitative research is the challenge high volumes of unstructured data poses. How does a researcher effectively and rapidly study high volumes of unstructured data in a comprehensive way without getting locked into their own biases?

The approach advocated by Cognitive Edge involves engaging the contributor of content in the analysis. In this effort we work towards distributing the cognitive load of analysis to a large population rather than concentrating the cognitive load on a small research team. The way this works is that a research team constructs a wide-scoping signification framework (see a new signification element, the triad, in one of Dave’s blogs) that is purposefully setup to reveal deep insights related to the domain of inquiry or objectives of a project. So for example if you had a project that had the general public contribute personal stories of their families experiences with the current economic crisis, you as a researcher might be interested in understanding the overall populations degree of hope, despair, fear, trust in the financial system, and possibly some other dimensions. After a respondent shares a story they then answer a series of specially designed signification questions (i.e. related in this example to hope, despair, fear, trust) about their story which not only interprets the content but adds layers of meaning that is not in the content itself. In this sense the contributor of the qualitative data contributes additional meaning to the content at the time of entry. As these interpretations aggregate across 100s if not 1000s of entries, emergent effects can be identified across large volumes of data using large volume data visualizations. In essence the meta-data (the question responses regarding the stories) provide insights into the collective perspective as defined by the general domain of inquiry - the example given being the collective experiences of families living through the financial crisis. The researcher now does not need to process 1000s of entries of unstructured data themselves, instead they leverage the meta-data contributed by the content contributor and dive into the content on an exception basis. This latter point is critical in minimizing the skewed interpretive effects of cognitive bias in the researcher. If a researcher is forced to move between visual patterns in meta data and then into content related to such patterns (or lack of patterns) then they have a better chance of disrupting pattern entrainment related to their cognitive bias. In addition to the visualization of patterns in the interpretive data we can also leverage statistics to identify trends and other patterns and rely on the content behind such statistics to add meaning to the numbers. In this sense the approach brings together both qualitative and quantitative aspects of research.

So back to the title of this post - “How do you make sense of 2000 ideas?” I summarize a recent project I worked on. In January an organization launched into an ambitious project of consulting across an employee base of approximately 4000 staff for ideas of ways to transform their business. The intent was to get all sorts of ideas from various contexts of the work environment from nearly every corner of the organization. The time to gather analyze and complete a final report on the effort was a mere 6 weeks. In less than 3 weeks we had over 2000 well thought out and articulated ideas contributed by over 750 respondents (nearly a 19% response rate). In the three weeks following that the meta-data was reviewed by a team of three project leads with the support of me using SenseMakerTM Explorer and other supplementary tools. A report was finalized and presented to the leadership team of this organization. In comparison to a similar effort by another part of the organization which about a year earlier gathered around 500 ideas from staff, their analysis effort took in excess of 6 months and was incredibly difficult since the interpretation of the raw data was put entirely onto a small team of researchers. Compared with 2000 ideas which were interpreted by over 750 individuals the distribution of the cognitive load was highly effective - not to mention that the contributors of the ideas have a far deeper sense of the meaning of what they wrote compared with what a researcher can make sense of only looking at their content and in most cases completely disconnected from their context.

So when you consider tapping into your employees, customers, or citizens on a mass consultation process don’t put the entire interpretive load on a small research team no matter how good they are at what they do. Distribute the cognitive load to the contributors in a way that gets deeper insights from the context of the content in the submissions. A process of inquiry that integrates engagement into the process will provide you as a leader the ability to quickly move forward and leverage your organizations readiness and willingness to transform, all according to a series of options they have qualified and analyzed for you. Oh and by the way the diversity and large range of perspectives that emerge in such data will force you to think differently and innovate -- so be prepared!

PS - the image in this post is from the 2000 idea data set (anonymized). The scatter plots are the interpretations of the ideas by the contributors themselves. Can you see the cluster emerging of ideas that are straight-forward and can be implemented immediately? (Hint: Graph in Row 1 Column 2)