Earl Mardle while generally agreeing with my post on IT education in schools takes issue with me over my statement that: In any complex system you can never replicate outcome, but you can replicate starting conditions. Now this is probably a conversation best held over a Sassy Red in The Brewery or brunch in Felix (my favorite cafe in Wellington) when I am out in New Zealand in December but I thought it best to give a response now as its an important issue.
Now his objections seem to be:
- That an any subsequent experiment the starting conditions are changed by awareness of the previous outcome
- Any second trip will ritually follow the first unless you consciously attempt not to.
Now remember we were talking about IT education in schools and the question of how to replicate an experiment. The normal practice is for a series of trials to be run, and where success is achieved to attempt to replicate success in other schools. My argument is that this fails in the main for two reasons: (i) the trial schools get more attention and often pre-select themselves and (ii) the context (social, historical etc) of each school is different in subtle (or less than subtle ways) and this disturbs the assumed contextual homogeneity of the experiment and roll out process. This isn't just schools by the way, it also applies in marketing and other areas.
My argument was that you could never replicate an outcome and Earl agrees with that; the disagreement is over the replication of starting conditions. Now I want to maintain my position, while pointing out that the phrase I used is an abbreviate of a complex set of arguments so misunderstanding is possible. Why do I do so?
For starters, it is very unlikely that each new school is going to be fully aware of what happened elsewhere, and there is no particular need for this to be the case. Even if they are aware they would not be able to replicate the path of the experiment. Here Earl's reference to Lorenz's discovery that minor changes have major impact on results contradicts his second argument as to ritualisation. It will take no conscious effort not to follow the same path, it will happen anyway. The intention is not to achieve the same outcome, but allow a contextual solution to emerge in each situation. We are not condemned just to be active and constructive, although that is no bad thing; we can learn.
Lets take another example to illustrate the point. I observe my parents as a child, and also other parents, when I am a parent myself. From that and other learning I know that some basics at the starting point of a child's development are critical. If you say something you should always follow through, always be there when they need you etc. etc. I don't believe that all children will end up the same, but I can learn. I don't have to always start from scratch.
So trying to exactly replicate the starting conditions might well be wasteful, it is in anyway impossible. However in the context of experiments and transfer of learning (the objectives of the post) one would not be attempting an exact replication, but rather to say something along the lines of XYZ put these things in place, had these resources and took these actions and things worked for them. Having done that, you move on work out the degree to which you can replicate those starting conditions, or where, given your local context to have to provide a substitute. You then take action and monitor, reinforcing good patterns, disrupting the bad ones. Now I summarised that quickly in the phrase: In any complex system you can never replicate outcome, but you can replicate starting conditions. As a summary I stand by it, hopefully now I have fleshed out the detail a bit hopefully Earl may modify his criticism! If not, then it is back to the Sassy Red.
Comments (6)
You can approximate the starting conditions, but you can't replicate them. The point is that even a seemingly insignificant difference in the starting conditions can change the outcome.
You don't get to create 'sameness' linguistically. If you say "Experiment A had 16 students" and "Experiment B had 16 students" you may appear to have 'replicated' the starting conditions, but the two experiments have, of course, different students (or the same students, but at different times). This is at *best* an approximation, and a very long way from replication.
Are approximations useful? Sure. But you should not be surprised to get results very different from what you gotr the first time. This - in my view - limits the usefulness of approximations.
Posted by Stephen Downes | October 18, 2007 4:17 PM
Posted on October 18, 2007 16:17
I'm not sure how well it fits with the theory, or your terminology Dave, but I wonder if there's a useful distinction here between the 'starting context', and the 'starting conditions'. It's of course nonsensical to try to replicate the starting context. There will be different people, a different physical environment, different weather, there might have been national media announcements since the last event that have affected people's mood (or in the case of a school announcements by the principal).
Starting conditions you can replicate because they're much simpler. I'm fairly sure that if I put a soccer ball on the ground near a group of 6 year old boys that some kind of 'soccer-like' behavior has a chance of emerging. It can be a different group of boys from last time I did it, different playing surface, different weather. I can also fairly easily set boundaries by saying things like "remember, no punching". If some 'soccer-like' behavior does emerge, I can wait to see if other interventions to stablise a 'soccer-like' pattern might be necessary. I might encourage two of the kids to take of their jerseys and make goal posts. There might end up being two sets of goals or just one. I might decide to say "remember, no hands", or I might wait to see if some form of rugby/soccer hybrid (or some other game entirely) emerges.
Now to Earl's objection to "In any complex system you can never replicate outcome, but you can replicate starting conditions". I think he's saying that you can't replicate the 'starting context' for the teacher, because the teacher has already experienced the original outcome and process. The teacher is changed. They are no longer in exploratory/experimenter mode, they are well on the path to process entrainment and ritualisation (an important survival tactic for overworked teachers).
For me it comes down to outcomes again. If the desired outcome is 'an exact replica of what happened last time' then the change in the teacher's starting context might lead them to impose structure to force the desired outcome. You might then end up with similar outwardly visible behavior as last time, but nothing like similar learning experiences taking place. I saw this often in my time at school, during my teacher training, and in the very brief time I actually spent teaching high school.
A good teacher (or parent, facilitator, manager) should however be able to let go of the exact, outwardly visible outcome, and focus on allowing the desired type of inward learning to take place. If the desired outcome was something like 'the kids will learn some more physical coordination, how to interact with each other in a game, and how to negotiate and stablise adaptive boundaries themselves' then as a teacher the fact that I've done something similar before shouldn't be a hindrance. It should make it more likely that if I can identify what the important starting conditions were (as distinct from the irrelevant contextual elements), then I should be able to have something vaguely within the range of the learning experiences I'm after, occur.
It's like good birthday party management. I've run 16 kids birthday parties and I've gotten fairly good at it. Not because I have a particular set of rituals or overt processes I follow, but because I know what to look for, when to intervene, and what kind of catalysts and boundaries to prepare in advance. The starting context is always different, but useful starting conditions can fairly easily be replicated.
Posted by Julian Carver | October 18, 2007 8:55 PM
Posted on October 18, 2007 20:55
OK lets make sure we are not misunderstanding each other.
I am not (Stephen) arguing that replicating the starting conditions will replicate the outcomes, the opposite in fact.
Let us remember the context here - taking an experiment which works in one school and scaling it. Not a teacher replicating an experiment of which they have been a part. Here if you replicate the starting conditions, then commence a series of safe-fail experiments and amplify or dampen the resultant patterns, then you can evolve towards a different, equally valuabel but contextually appropriate future. I am arguing here for replication of learning as a process rather than was has been learnt as a thing.
Julian - I like your conclusion and the context/conditions distinction. It's very useful
Posted by Dave Snowden | October 18, 2007 10:28 PM
Posted on October 18, 2007 22:28
Hi Dave
The issues of starting contexts/circumstances and outcomes in schools and other public services is something I feel very strongly about. Having worked with a number of different schools (primary, secondary and special) and different public services then I would agree that one cannot replicate outcomes from one establishment to another, and suggest it is impossible (and a waste of money, staff time and energy) to replicate starting conditions. Each school/service in/with which I have worked has been different in key respects: the location of the building in relation to its 'client' population; the physical structures and layout of the building; the individuals who are part (both immediate and peripheral) of that community; the stories that are told and circulated in/about that community; and the histories, circumstances, values and beliefs of those individuals. The variety in even a very small school or public service is significant - for a large secondary school, Council/Health service division, mind boggling.
Unfortunately the current (British) Government obsession with 'Evidence Based Practice ('What Works) and 'Best Practice' comfortably ignores all of these factors: so we get the 'if it worked in New Zealand (Family Group Meetings) then its the answer to youth offending in Bristol' argument - followed by the, 'if you haven't made it work (reduce offending) in Bristol there's something wrong with you and your organisation'.
Surely a much more productive approach (but one that generates fewer facts and soundbites for politicians) is to have conversations with a range of stakeholders about how we might prepare our children for a future that is uncertain. For those who haven't seen it, take a look at Shift Happens (link to YouTube below) - for me this goes beyond teaching IT in schools, to the far deeper question of how can we encourage/enable self-organisation that will underpin the availability of learning opportunities for children and young people?
http://www.youtube.com/watch?v=pMcfrLYDm2U
Best Wishes
David
Posted by Dave Hoyle | October 19, 2007 1:03 PM
Posted on October 19, 2007 13:03
The subject of Evaluation Research is massive, with a large body on educational evaluation. I'd have to explore loft/shed/garage to get a proper slice of literature, but good starting points are:
Realistic Evaluation by Pawson and Tilley,
Utilisation-focused evaluation by Michael Quinn Patton,
The Art of Case Study Research by Robert E Stake.
There are standards for the conduct of such evaluations, but don't have the reference to hand. Can go exploring if that would help.
Posted by Brian Sherwood Jones | October 19, 2007 7:35 PM
Posted on October 19, 2007 19:35
Well Dave, great to see you frequent my old home town (Wellington). And to see you are running an accreditation course there in December.
However, I can't believe you ended up rooting for the French against the All Blacks! And yes we are also in National shock / grief.
From CA,
Tony
Posted by Anthony Kortens | October 22, 2007 11:27 PM
Posted on October 22, 2007 23:27