Scaling in complex systems

I wrote a series of posts about scaling over three year ago which in part addressed some of the perversities of methods focused on accreditation revenue to which I referred yesterday. They were important posts and sections will be dumped in Scrivener over the next months as I get ready for an intense booking writing week in a remote cottage on Ynys Môn this January. Naturally I have done a lot more thinking since then but the fundamentals remain: you don’t scale a complex system by aggregation or imitation but by decomposition to an optimal level of granularity followed by recombination (possibly exaptive) in context: complexity is about how things connect far more than what the things are. I remember an argument about this in IBM days when I had finished working through a radical new organisational model in which the role of IBM would be to centre a network rather than own resource. It was rejected on the basis that scale was perceived in terms of size and number of reports, in turn linked to hierarchical and aggregative/reductionist ideology.

The so-called scaled frameworks in Agile made, and continue to make similar mistakes with varying degrees of absurdity. The worst simply aggregates everything that has ever been called Agile into a complicated flow diagram, reducing complexity to an over-constrained structure. This is scaling by dumbing down to a lowest common denominator approach; it gives an impression of order by constraining practice to the explicit. As such it mimics the errors made in BPR which were magnified in Sick Stigma to the point of doctrine. The net result of all reductionist approaches to process management was the emergence of high energy wastage and stress as humans worked around the system to make it work despite itself; effectiveness was sacrificed on the altar of superficial efficiency. Interestingly in twitter encounters with some advocates of the SAFe Borg yesterday this (the idea that you work around something to make it work) was put forward as a defence – good consultants/coaches cherry pick what they think will work and modify it in practice while maintaining the pretence of the over-simplification they think is necessary to sell to senior IT Management. I think the phrase here is trying to force a virtue out of a vice.

To illustrate the point let’s take organisational forms. We know that the behavioural characteristics of teams of five, fiveteen and one hundred and fifty are very different. You can’t take the practices and interdependencies of a small team and replicate them in a large team. In a small team a level of intimacy and knowledge of the characteristics of other team members allows smooth coupling and pre-emptive actions to mitigate faults and weakness. In a large team we become increasingly dependent on the explicit statements of qualifications and competences and less on the tacit understanding of capability in different contexts; compliance and conformity are privileged over anticipation and adaption. Taking a complex perspective we would create variable expectations of different identity clusters (people can belong to different identities) with management and monitoring of the nature of interactions and emergent stabilities that arise from those interactions. This allows a large system to grow very quickly and at times in novel and unexpected ways as a series of situated networks. Think of a Portuguese Man of War which is a symbiosis of different creatures which assemble in a specific context, or of Slime Moulds which again change their nature under conditions of starvation; in software Object Orientation and Micro-services allow novel combinations to emerge if properly architected. It’s not that we don’t have the models to use here, its just that we are stuck in engineering diagrams.

Alicia Juarrrero famously coined the phrase like bramble bushes in a thicket to describe a complex adaptive system and it is a title Kurtz and I used in this paper which represents early thinking on organisational design and networks. In a complex system you need to define your identity focal points and manage your connections, rather then get out the flow charting template and create excessive structure. Bramble bushes bear fruit, but also prick the unwary; structured flow diagrams are safe in the sense of the absence of novelty, the absence of risk and the absence of imagination.