Tag Archives: complexity

Complexity isn’t (at least in enterprise IT)

There’s a lot of talk these days about complexity in enterprise IT. The heterogeneous solutions we’re building today seem more complex than the monolithic solutions of the past. But are they really? I’ll grant you that a lot of what is being built at the moment is complicated. Complex though? I don’t think so. The problem is that we’re building new things in old ways when we need to be building new things in new ways.

I’ve always used a simple rule of thumb when thinking about complexity. Some folk like to get fancy with two and three dimensional models that enable us to ascribe degrees of complexity to problems. While I find these models interesting, my focus has always been on how do I solve the problem in front of me. What is the insight that will make the hard easy? For me, one simple distinction seems to provide the information I need. Is a solution complex? Or is it complicated?

Something is complicated if the model we use to understand the problem requires patches, exceptions to make it work. The model might be simple and well understood, but we’re forced to patch the model for it to succeed when confronted by the real world; we’re adding epicycles. It’s not a complex system, but it is complicated.

Adding epicycles didn't manage to keep the earth at the centre of the universe
Adding epicycles didn't manage to keep the earth at the centre of the universe

On the other hand, something is complex if it’s difficult to develop a consistent model for the problem. While we might have a well understood model, it’s definitely not simple, requiring a great deal of academic and tacit knowledge to understand. There’s no epicycles, but there are a large number of variables involved, and their interactions are often non-linear.

While this binary separation might not be strictly true (the complicated can sometimes be complex), I find that the truly complex problems are rare enough that the rule of thumb is useful most of the time. After all, that’s what a rule of thumb is. The few times that it breaks down, experience comes to the rescue.

Distinguishing between the complex and the complicated is not hard; just look for the epicycles. Planning engines – such as material planning or crew scheduling – are a good example of a complex solutions. Business processes management is a good example of a complicated solution. Psi calculus – the model at the heart of a modern BPM engine – is well understood and BPM engines work as described. However, managing business exceptions is a mess as support for them is tacked on rather than an inherent part of the model. Smashing together psi calculus, transactions and number of other models has resulted in epicycles.

Most of the problems we’re seeing in enterprise IT are complicated, but not complex. Take the current efforts to create IT estates integrating SaaS and public cloud with more traditional enterprise IT tools, such as on-premesis applications and BPO. Conventional approaches to understanding and planning IT estates are creaking at the seems. The model – the enterprise integration patterns – which we’ve used for so long is well understood, but it’s creaking at the seams as we bolt on more epicycles to cope with exceptions as they arise.

Dr. Khee Pang
Dr. Khee Pang

A great piece of advice from a former lecturer of mine always comes to mind in situations like this. As I’ve mentioned before:

If you don’t like a problem, then change it.
KK Pang

Our solution is complicated because we’re trying to solve the wrong problem. We need to change it.

The problem with BPM is that business exceptions are not exceptional, but are simply alternative ways of achieving the same goals. To resolve the epicycles we need to shift the problems centre of gravity, moving the earth from the centre of the universe to a more stable orbit. If business exceptions are not exceptional, then we should simply consider them as different business scenarios, and use a scenario based approach to capturing business processes. The epicycles then melt away.

I think we can use a similar approach to help us with the challenges we’re seeing in today’s IT estates, the same challenges which are trigger some of the discussion on complexity. The current approach to planning and provisioning IT is data centric; most applications are, after all, just large data management engines. This data-centric approach is forcing us to create epicycles as we attempt to integrate SaaS, cloud, and a whole raft of new tools and techniques. The solution is to move from a data-centric approach, to decision-centric approach. But that’s a different blog post.

We’re making our lives too complicated

Has SOA (Service Oriented Architecture) finally jumped the shark? After years of hype and failed promises, SOA seems to be in trouble. In a few short months it’s gone from IT’s great savour to something some people think is better forgotten.

The great promise of SOA was to deliver an IT estate which is more agile and cost effective than was possible with other, more conventional, approaches. By breaking our large problems into a set of much smaller ones, we would see more agility and a low total cost of ownership. The agility would come from the more flexible architecture provided by SOAs many moving parts. A lower cost of ownership would come from reuse of many of these moving parts. Many companies bought into this promise, and started major SOA transformation programs to “SOA enable their business”. Once the program of work was delivered they would have a shiny new flexible, and cost effective IT estate. The business would be thrilled, and the old tensions between business and IT would just melt away. More often than not the business wasn’t thrilled as the program failed to deliver the promised benefits.

The problem, it seems, is that we’re focused on creating cathedrals of technology. Cathedrals were the result of large bespoke development efforts. The plans often consisted of only a rough sketch on a scrap of paper, before a large number of skilled craftsmen were engaged. The craftsmen broke the problem into many small parts that were then laboriously assembled into the final structure, often adjusting the parts to fit in the process. While this process created a number of spectacular buildings, the journey from initial conception to completed build was long and challenging.

The lack of engineering pragmatism frequently resulted in cathedrals collapsing before they were finished, often multiple times. The reason we know that a flying buttress worked was because it hadn’t failed, yet. People died when a structure collapsed, and there was no way of telling if the latest version of the structure was about to collapse. The lengthy development process often lasted generations, passing through the stewardship of multiple architects with no clear end in sight. Many cathedrals, such as the one in New York, are still considered unfinished.

A lot of SOA projects give off a strong smell of cathedral. They are being constantly re-architected—while still in development—to cope with the latest business exception or demand. When they’re introduced to the hard reality of supporting solutions bits of them collapse and need to be rebuilt to support our new (and improved) understand of what will be demanded of them. And, finally, many of them are declared “finished” even though they are never fully baked, just so we can close that chapter in our company’s history and move onto the next problem.

Modern approaches to building construction take a different approach. A high level plan is created to capture the overall design for the building. The design is then broken into a small number of components, with the intention for using bespoke craftsmen for the fine details that will make the building different, while leveraging large, commoditized, pre-fabricated components for the supporting structures that form the majority of the building. Construction follows a clear timetable, with each component—from the largest pre-fabricated panel through to the smallest detail—integrated into the end-to-end solution as it is delivered. Complexity and detail were added only where needed, with cost effective commoditized approaches minimizing complexity elsewhere. A clear focus on the end goal is maintained throughout the effort, while clear work practices focused on delivering to the deadline ensure that the process was carried out with a minimum of fuss (and no loss of life).

The problem, it seems, is that we’re confusing agility with flexibility. The business is asking for agility; the world is changing faster than ever and the business needs to be in a position to react to these changes. Agility, or so our thinking goes, requires flexibility, so to provide a lot of agility we need to provide a lot of flexibility. Very soon we find ourselves breaking the IT estate (via our favorite domain model) into a large number of small services. These small parts will provide a huge amount of flexibility, therefore problem solved!

This misses the point. Atomizing the business in this way creates overhead, and the overhead soon swamps any benefit. The effort to define all these services is huge. Add a governance process—since we need governance to manage all the complexity we created—and we just amplify the effect of this overhead. Our technically pure approach to flexibly is creating too much complexity, and our approach to managing this complexity is just making the problem worst.

We need to think more like the architect of the modern prefabricated building. Have a clear understanding of how the business will use our building. Leverage prefabricated components (applications or SaaS) where appropriate; applications are still the most efficient means of delivering large, undifferentiated slabs of functionality. And add complexity only in those differentiating areas where it is justified, providing flexibility only where the business needs. In the end, creating good software is about keeping it simple.  If it’s simple, it gets done quickly and can be maintained more readily.

Above all, favor architectural pragmatism over architectural purity. The point of the architecture is to support the business, not to be an object of beauty.