Tag Archives: Agentis

BPM is not a programming challenge

Get a few beers into a group of developers these days and it’s not uncommon for the complaints to start flowing about BPM (Business Process Management). BPM, they usually conclude, is more pain than it’s worth. I don’t think that BPM is a bad technology, per se, but it does appear to be the wrong tool for the job. The root of the problem is that BPM is a handy tool for programming distributed systems, but the challenge of creating distributed systems is orthogonal to business process execution and management. We’re using a screw driver to belt in a nail. It’s more productive to think business process execution and management as a (realtime) planning problem.

Programming is the automation of the known. Take as stable, repeatable process and automate it; bake the process into silicone to make it go fast. This is the same tactic that I was using back in my image processing days (and that was a long time ago). We’d develop the algorithms in C, experiment and tweak until they were right, and once they were stable we’d burn them into an ASIC (Application-Specific Integrated Circuit) to provide a speed boost. The ASICs were a lot faster than the C version: more than an order of magnitude faster.

Programmers, and IT folk in general, have a habit of treating the problems we confront as programming challenges. This has been outstandingly successful to date; just try and find a home appliance or service that doesn’t have a programme buried in it somewhere. (It’s not an unmitigated success though, such as our tumble drier is driving us nuts if its overly frequent software errors.) It’s not surprising that we chose to treat business processes automation and management as a programming problem once it appeared on our radar.

Don’t get me wrong: BPM is a solid technology. A friend of mine once showed my how he’d used his BPM stack to test its BPEL engine. As side from being a nice example of eating your own dog food, it was a great example of using BPEL as a distributed programming tool to solve a small but complex problem.

So why do we see so many developers complaining about BPM? It’s not the technology itself: the technology works. The issue is that we’re using it to solve problems that it’s not suited for. The most obvious evidence of this is the current poor state of BPM support for business exception management. We’ve deployed a lot of technology to support exception management in business processes without really solving the problem.

Managing business exceptions is driving the developers nuts. I know of one example where managing a couple of not infrequent business exceptions was the major technical problem in a very significant project (well into eight figures). The problem is that business exceptions are not from the same family of beasts as programming exceptions. Programming exceptions are exceptional. Business exceptions are just a (slightly) different way to achieve the same goal. All our compensating actions and exception stacks just get in the way of solving the problem.

On PowerPoint, anything can look achievable. The BPMN diagram we shared with the business was extremely elegant: nice sharp angles and coloured bubbles. Everyone agreed that it was a good representation of what the business does. The devil is in the details though. The development team quickly becomes frustrated as they have to deal with the realities of implementing a dynamic and exception rich business processes. Exceptions pile up on top of exceptions, and soon that BPMN diagram covers a wall, littered as it is with branch and join operations. It’s not a complex process, but we’ve made it incredibly complicated.

Edward Tufte's take on explaining complex concepts with PowerPoint
A military parade explained, a la PowerPoint

We can’t program our way out of this box, trying to pile on more features and patches. We can rip the complications out – simplifying the process to the point that it becomes tractable with our programming tools (which is what happened in my example above). But this removes all the variation which which makes the processes so valuable. (This, of course, the dirty secret of LEAN et al: you’re trading flexibility for cost saving, making your processes very efficient but also very fragile.)

Or we can try solving the problem a different way.

Don’t treat the automation of a business processes as a programming task (and I by this I mean the capture of imperative instructions for a computer to execute, no matter how unstructured or parallel). Programming is the automation of the known. Business processes, however, are the management and anticipation of the unknown. Modelling business processes should be seen as a (realtime) planning problem.

Which comes back to one of my common themes: push vs pull models, or the importance of what over how. Or, as a friend of mine with a better turn of phrase puts it, we need to stop trying to invent new technologies and work out how to use what we already have more effectively. Rather than trying to invent new technologies to solve problems that are already well understood elsewhere, pushing the technology into the problem, a more pragmatic approach is to leverage that existing understanding and then pull in existing technologies as appropriate.

Planning and executing in a rapidly changing environment is a well understood problem. Just ask anyone who’s been involved with the military. If we view the management of a business processes as a realtime planning problem, then what were business exceptions are reduced to simply alternate routes to the same goal, rather than a problem which requires a compensating action.

Battle of Gaugamela (Arbela) (331BC)
Take that hill!

One key principle is to establish a clear goal – Take that hill!, or Find that lost shipment! – articulate the tactics, the courses of action we might use to achieve that goal, and then defer decisions on which course of action to take until the decision needs to be made. If we commit to a course of action too early, locking in a decision during design time, then it’s likely that we’ll be forced to manage the exception when we realise that we picked the wrong course of action. It’s better to wait until the moment when all relevant information and options are available to us, and then take decisive action.

From a modelling point of view, we need to establish where are the key events at which we need to make decisions in line with a larger strategy. The decisions at each of these events needs to weigh the available courses of action and select the most appropriate, much like using a set of business rules to identify applicable options. The course of action selected, a scenario or business process fragment, will be semi independent from the other in the applicable set, as it addresses a different business context. Nor can the scenario we pick cannot be predetermined, as it depends on the business context. Short and sharp, each scenario will be simple, general and flexible, enabling us to configure it for the specific circumstances at hand, as we can’t anticipate all possible scenarios. And finally, we need to ensure that the scenarios we provide cover the situations we can anticipate, including the provision of a manual escape hatch.

Goals, rules and process: in that order. Integrated rather than as standalone engines. Pull pull these established technologies into a single platform and we might just be closer to a BPM solution inline with what we really need. (And we know there is nothing new under the sun, as this essentially a build on Jim Sinurs rules-and-process argument, and borrows a lot from STRIPS, PRS, dMARS and even the work I did at Agentis.)

As I mentioned at the start of this missive, BPM as a product category makes sense and the current implementations are capable distributed programming tools. The problem is that business process management is not a distributed programming challenge. Business exceptions are not exceptional. I say steal a page from the military strategy book – they, after all, have been successfully working on this problem for some time – and build our solutions around ideas the military use to succeed in a rapidly changing environment. Goals, rules and processes. The trick is to be pragmatic, rather than dogmatic in our implementation, and focus on solving the problem rather then trying to create a new technology.

Innovation and the art of random

A little while ago I was invited to speak at an event, InnoFuture, which, for a mixture of reasons, didn’t end up happening. The theme for the event was Ahead of the trends — the random effect. My take on it was that innovation is not random, it’s just happening faster than you can process, and that ideas are commoditized making synthesis, the creation of new solutions to old problems, what drives innovation. I was pretty happy with the outline I put together for my talk, that I ended up reusing the content and breaking it into three blog posts, rather than letting it go to waste.

Innovation seems to be the topic of the day. Everyone seems to want some, thinking that it’s the secret sauce which will help them (or their company) bubble to the top of the heap. The self help and consulting communities have responded in force, trying to bottle lightening or package the silver bullet (whichever metaphor you prefer).

It was in this environment that I was quite taken by the topic of a recent InnoFuture event when I was asked to speak.

Ahead of trends — the random effect.
When a concept becomes a trend, you are a not the leader. How to tap into valuable ideas for products, services and communication before they are seen as trends, when they are just … random? Albert Einstein said that imagination is more important than knowledge. Let’s open the doors and let the imagination in for it seems that in the current crisis, the right brain is winning and we may be rationalized to death before things get better.

I’ve never seen the random effect, though I have been delightfully surprised when something unexpected pops up. Having been involved in a bunch of companies and projects that, I’m told, where innovative, I’ve always thought innovation was not so much random, as the result of obliquity. What makes it seem random is the simple fact that your are not aware of the intervening steps from interesting problem through to novel solution.

I figured I’d mash together a few ideas that capture this thought, and provide some (hopefully) sage advice based on what I do to deal with random. I ended up selecting:

  • John Boyd on why rapidly changing environments are confusing,
  • Peter Drucker‘s insight that insight (the tacit application of knowledge) is not a transferable good,
  • the struggle for fluency that we all go through as we learn to read,
  • John Boyd (again, but then he had a lot of good ideas) on the need for synthesis,
  • KK Pang (and old lecturer of mine) on the need to view problems from multiple contexts,
  • the need to follow a consistent theme of interest as the only tractable way of finding interesting problems to solve, and
  • my own experiences in leveraging a network of like and dissimilar minds as a way of effectivly out-sourcing analysis.

The result was called Of snow mobiles and childhood readers: why random isn’t, and how to make it work for you. I ended up having far to much content to fill my twenty minute slot, so it’s probably for the better that the event didn’t go ahead, as it would have taken a lot of time to cut it down.

Given that I had a fairly well developed outline, I decided to make it into a series of blog posts (plus my slides these days don’t have a lot of text on them, so if I just dropped the slides online they wouldn’t make any sense). The blog posts ended up breaking down this way:

  1. Innovation should not be the race for the new-new thing.
    Points out that innovation only seems random, unexpected, as you don’t see the intervening steps between a problem and new solution, and that innovation is the result of many small commoditized steps. This ties into one of my earlier posts of dealing with the speed of change.
  2. The role of snowmobiles in innovation.
    Argues that ideas are a common commodity, and that the real challenge with innovation is synthesis rather than ideation.
  3. Childhood readers and the art of random.
    Argues that the key to innovation is to find interesting problems to solve, and suggests that the best approach is to be fluent in a range of domains (sectors, geographies, activities, …) to provide a broader perspective, focus on a line of inquiry to provide some structure, and build a network of people with complimentary interests, providing you with the time, space and opportunity to focus on synthesis.

I expect that these are more productive if taken as a whole, rather than individual posts.

If you look at the path I’ve charted over my career then this is the approach I’ve taken, and my topic of choice is how people communicate and decide as a group, leading me to John Boyd, Cicero, human-computer interaction, agent technology, biology (my thesis was mathematically modelling nerves in a cat), and so on.

I still have the slides, so feel free to contact me it you’re interested in my presenting all or part of this topic.