When I was a statistician, I had almost complete professional freedom, as the people I worked for or with did not consider themselves qualified to judge whether the approaches I took were the right ones. In the world of education, the opposite happens – as everyone has gone to school for more than a decade, everyone considers him- or herself an expert on how it should be done, and therein lies the rub. Many an intervention is tried, and even implemented on large scale, because it sounds like a good idea, with very little evidence as to its suitability for the particular context.
So the Foundation I work for set out to generate evidence to support solutions in all our programmes. That turned out to be easier said than done in something like a bursary programme. The sample sizes required to achieve acceptable power and discrimination using traditional statistical methods were simply prohibitive in terms of cost. All the ethical issues familiar in social research presented themselves; for example, how can we not provide the support we believe can make a difference to the bursars’ success just for the sake of having a control group in the experiment? And if we see something is not working, how can we not intervene just to get good data?
The approach to use safe-fail probes appears appropriate and solves some of the problems, but raises others. What we do know about students is that they often don’t realize they are in trouble until it’s too late, and that for many it’s a self-esteem issue to not ask for help, so expecting them to voluntarily join a support programme may not meet their real needs. Although the combination of narratives and quantitative data available from SenseMaker indeed convinces powerfully, the challenge still is to collect the narratives from a diverse enough population to see how the conclusions hold across different contexts before one can hazard any recommendations.
Then we get to the philosophical aspects of what constitutes convincing evidence for policy purposes. When I refer to evidence-based policy making, I often get a most cynical response. Case studies in the medical world show that a strong emphasis on data can lead to over-diagnosis and cause more problems than it solves.
So to get beyond implementing education policies just because they have worked elsewhere or seem like good ideas is not a simple problem; anyone who has experience with it or has an interest in grappling with it is most welcome to help us think about how to do it.
Comments (2)
While we in public health consider ourselves an evidence-based discipline, this is unfortunately "an aspirational goal, especially in the critical area of interventions, where we have been woefully unsuccessful when behavior change is required.
A recent review article by Thacker et al estimated that only ~5% of public health interventions are evidence based. While I think he was too conservative, his point is a good one and I am sure he is in the right order of magnitude.
CDC's Community Guidelines and the global Campbell Collaboration are slowly beginning to fill these voids.
Most of us in public health are hard quants, so the behavioral science required to change behaviors leaves us feeling vaguely uneasy. This is changing and needs to change dramatically if we are to move past elegantly chronicling our failures.
Posted by Mark White | March 14, 2010 6:01 PM
Posted on March 14, 2010 18:01
Your blog post raises many familiar issues for me - I am working within a large Department of Education (K-12) and am frustrated by the lack of any attempt to assess the value of programs. A common view within the department is that it is impossible to assess the impact on student outcomes and therefore not worth the cost/effort. I don't completely buy that as the cost of implementing many programs are massive in terms of up front costs as well as ongoing maintenance therefore the cost of not assessing the value provided is equally significant.
I am just starting to apply narrative based impact assessment approaches as a way of 'measuring' the value of programs. I am currently commencing a project to engage parents in the learning outcomes of students and encourage interaction and information exchange between teacher-parent-student. Although the objective is to improve learning outcomes of students we are going to use narrative based data to measure the impact that the project has on increasing parental engagement. (We will be gathering data from parents, teachers and students to assess this) This is based on research that links parental engagement to improved student learning outcomes. It is early days for us - however looking forward to demonstrating a sustainable method for the department.
I'd be interested in following your efforts and exchanging ideas.
Posted by David Green | March 15, 2010 3:00 PM
Posted on March 15, 2010 15:00