Tag Archives: Process management

BPM is not a programming challenge

Get a few beers into a group of developers these days and it’s not uncommon for the complaints to start flowing about BPM (Business Process Management). BPM, they usually conclude, is more pain than it’s worth. I don’t think that BPM is a bad technology, per se, but it does appear to be the wrong tool for the job. The root of the problem is that BPM is a handy tool for programming distributed systems, but the challenge of creating distributed systems is orthogonal to business process execution and management. We’re using a screw driver to belt in a nail. It’s more productive to think business process execution and management as a (realtime) planning problem.

Programming is the automation of the known. Take as stable, repeatable process and automate it; bake the process into silicone to make it go fast. This is the same tactic that I was using back in my image processing days (and that was a long time ago). We’d develop the algorithms in C, experiment and tweak until they were right, and once they were stable we’d burn them into an ASIC (Application-Specific Integrated Circuit) to provide a speed boost. The ASICs were a lot faster than the C version: more than an order of magnitude faster.

Programmers, and IT folk in general, have a habit of treating the problems we confront as programming challenges. This has been outstandingly successful to date; just try and find a home appliance or service that doesn’t have a programme buried in it somewhere. (It’s not an unmitigated success though, such as our tumble drier is driving us nuts if its overly frequent software errors.) It’s not surprising that we chose to treat business processes automation and management as a programming problem once it appeared on our radar.

Don’t get me wrong: BPM is a solid technology. A friend of mine once showed my how he’d used his BPM stack to test its BPEL engine. As side from being a nice example of eating your own dog food, it was a great example of using BPEL as a distributed programming tool to solve a small but complex problem.

So why do we see so many developers complaining about BPM? It’s not the technology itself: the technology works. The issue is that we’re using it to solve problems that it’s not suited for. The most obvious evidence of this is the current poor state of BPM support for business exception management. We’ve deployed a lot of technology to support exception management in business processes without really solving the problem.

Managing business exceptions is driving the developers nuts. I know of one example where managing a couple of not infrequent business exceptions was the major technical problem in a very significant project (well into eight figures). The problem is that business exceptions are not from the same family of beasts as programming exceptions. Programming exceptions are exceptional. Business exceptions are just a (slightly) different way to achieve the same goal. All our compensating actions and exception stacks just get in the way of solving the problem.

On PowerPoint, anything can look achievable. The BPMN diagram we shared with the business was extremely elegant: nice sharp angles and coloured bubbles. Everyone agreed that it was a good representation of what the business does. The devil is in the details though. The development team quickly becomes frustrated as they have to deal with the realities of implementing a dynamic and exception rich business processes. Exceptions pile up on top of exceptions, and soon that BPMN diagram covers a wall, littered as it is with branch and join operations. It’s not a complex process, but we’ve made it incredibly complicated.

Edward Tufte's take on explaining complex concepts with PowerPoint
A military parade explained, a la PowerPoint

We can’t program our way out of this box, trying to pile on more features and patches. We can rip the complications out – simplifying the process to the point that it becomes tractable with our programming tools (which is what happened in my example above). But this removes all the variation which which makes the processes so valuable. (This, of course, the dirty secret of LEAN et al: you’re trading flexibility for cost saving, making your processes very efficient but also very fragile.)

Or we can try solving the problem a different way.

Don’t treat the automation of a business processes as a programming task (and I by this I mean the capture of imperative instructions for a computer to execute, no matter how unstructured or parallel). Programming is the automation of the known. Business processes, however, are the management and anticipation of the unknown. Modelling business processes should be seen as a (realtime) planning problem.

Which comes back to one of my common themes: push vs pull models, or the importance of what over how. Or, as a friend of mine with a better turn of phrase puts it, we need to stop trying to invent new technologies and work out how to use what we already have more effectively. Rather than trying to invent new technologies to solve problems that are already well understood elsewhere, pushing the technology into the problem, a more pragmatic approach is to leverage that existing understanding and then pull in existing technologies as appropriate.

Planning and executing in a rapidly changing environment is a well understood problem. Just ask anyone who’s been involved with the military. If we view the management of a business processes as a realtime planning problem, then what were business exceptions are reduced to simply alternate routes to the same goal, rather than a problem which requires a compensating action.

Battle of Gaugamela (Arbela) (331BC)
Take that hill!

One key principle is to establish a clear goal – Take that hill!, or Find that lost shipment! – articulate the tactics, the courses of action we might use to achieve that goal, and then defer decisions on which course of action to take until the decision needs to be made. If we commit to a course of action too early, locking in a decision during design time, then it’s likely that we’ll be forced to manage the exception when we realise that we picked the wrong course of action. It’s better to wait until the moment when all relevant information and options are available to us, and then take decisive action.

From a modelling point of view, we need to establish where are the key events at which we need to make decisions in line with a larger strategy. The decisions at each of these events needs to weigh the available courses of action and select the most appropriate, much like using a set of business rules to identify applicable options. The course of action selected, a scenario or business process fragment, will be semi independent from the other in the applicable set, as it addresses a different business context. Nor can the scenario we pick cannot be predetermined, as it depends on the business context. Short and sharp, each scenario will be simple, general and flexible, enabling us to configure it for the specific circumstances at hand, as we can’t anticipate all possible scenarios. And finally, we need to ensure that the scenarios we provide cover the situations we can anticipate, including the provision of a manual escape hatch.

Goals, rules and process: in that order. Integrated rather than as standalone engines. Pull pull these established technologies into a single platform and we might just be closer to a BPM solution inline with what we really need. (And we know there is nothing new under the sun, as this essentially a build on Jim Sinurs rules-and-process argument, and borrows a lot from STRIPS, PRS, dMARS and even the work I did at Agentis.)

As I mentioned at the start of this missive, BPM as a product category makes sense and the current implementations are capable distributed programming tools. The problem is that business process management is not a distributed programming challenge. Business exceptions are not exceptional. I say steal a page from the military strategy book – they, after all, have been successfully working on this problem for some time – and build our solutions around ideas the military use to succeed in a rapidly changing environment. Goals, rules and processes. The trick is to be pragmatic, rather than dogmatic in our implementation, and focus on solving the problem rather then trying to create a new technology.

Accelerate along the road to happiness

Our ability to effectively manage time is central to success in today’s hype-competitive business environment. The streamlined and high velocity value-chains we’ve created are designed to invest as little time (and money) as possible in unproductive business activities. However, being fast, being good at optimizing our day-to-day operations, is no longer enough. We’ve reached a point where managing the acceleration of our business—the ability to change direction, redeploy resources to meet new opportunities more rapidly than our competition—is the driver for best in category performance. If we can react faster than our competition then we can capitalize on a business opportunity (or disruption, as they are often the same) and harvest any value the opportunity created.

Time is our overarching business driver at the moment. We hope to be the first to approve a mortgage, capturing the customer before our competitors have even responded to the original application. We strive to be first to market with a new portable music device (Walkman or iPod), establishing early mover advantage and taking the dominant position in the market. Or we might simply want to quickly restore essential services—power, gas or water—to our customers, as they have become intensely dependent upon them. Globalization has leveled the playing field, as we’re all working from the same play book and leveraging the same resources. The most significant factor for success in this environment is the ability to execute faster than our competition—harvesting the value in an opportunity before they can.

This focus on time is a recent phenomena. Not long ago, no further back than the early nineties, we were more concerned with mass. The challenge was too get the job done. Keep the wheels turning in the factories. Keep the workers busy in their cubicles. Time is money, so we’re told, and we need to ensure that we don’t waste money by laying idle. Mass was the key to success—ensuring that we had enough work to do, enough raw materials to work on, to keep our business busy and productive.

When mass is the focus, then bigger is better. This is a world where global conglomerates rule, as size is the driver for success. Supply chains were designed so that enough stuff was available right next to the factory, where supply can be ensured, that the factory would never run out of raw materials and grind to a halt. Whether shuffling paperwork or shifting widgets, the ability to move more stuff around the business was always seen as an improvement.

This is also the world that created a pile of shipping containers too behold in the Persian Gulf, during the Gulf War in the early nineties. With no known destination, some containers couldn’t be delivered. Without a clear understanding of where they came from, others couldn’t be returned. A few of these orphaned containers were opened in an attempt to determine their destination or origin; however the sweltering Arabian sun was not kind to their contents, which included items such as raw poultry, so a stop was soon put to that. The containers just kept piling up. 22,000 of 50,000 containers simply became invisible, collecting in a pile that went by the jaunty name of Iron Mountain.

Iron Mountain: 22,000 containers that became invisible
Iron Mountain: 22,000 containers that became invisible

Our answer was to stop focusing on mass, on having enough stuff on hand to keep the wheels of industry turning. We have to admit that Iron Mountain proves that we could move sufficient mass. The next challenge was to ensure that materials arrived at just the right time for them to be consumed by the business. We moved from worrying about mass, to managing velocity.

Total quality management and process improvement efforts finally found their niche. LEAN and Six Sigma rolled through the business landscape ripping cost out businesses where-ever they went. Equipped with books on Toyota’s Production System and kanban cards, we ripped excess material from the supply chain. Raw materials arrive just-in-time, and we avoid the costs associated with storing and handling vast warehouses of material, as well as the working capital tied up in the stored material itself. Quality went up, process cycle times shrunk, and the pace of business accelerated. Much like the tea clippers from China in the 1800s, with the annual race to get the first crop back to London for the maximum profit (with skipper paid a profit share as an incentive along with their salary), we’re focused on cranking the handle of business as fast as possible.

Zara, a fashion retailer, is the poster child for this generation of business. The fashion industry is built around a value-chain that tries to push out regular product updates, beating up demand via runway shows and media coverage to support a seasonal marketing cycle. Zara takes a different approach, tracking customer preferences and trends as they happen in the stores and trying to deliver an appropriate design as rapidly as possible, allowing customer demand to pull fashion. By focusing on responding to customer demand, wherever it is, Zara has built an organization designed too minimize time from design to marketed product. For example, onshore, high-tech, agile production is preferred to low-tech but low cost, offshore production which involves long production delays. Zara takes two weeks to take a product to market, where the industry average is six months; the lifetime of Zara’s products is measured in weeks, rather than months; and the products offered in each store are tailored to the interests of the community it serves rather than a long term marketing plan.

The change in product life-cycle has created a material change to customer buying habits. Traditionally customers’ will visit a fashion store a few times a year to see what a new season brings. There is no real pressure to buy in any particular visit, as they know they can return to buy the same garment later. Zara, however, with it’s dramatically shortened product cycles, drives different behavior. Customer visit more often, as they can expect to see a new range each visit. They are also more likely too buy, as they know that there is little chance of the same garment being available the next time. This approach has made Zara the most profitable arm of Inditex, a holding company of eight retail brands, and one of the biggest success stories in Spanish business.

The dirty secret of high velocity, lean businesses is that they are fragile: small disturbances can create massive knock-on effects. As we’ve ripped fat from the value chain, we’ve also weakened its ability to react to, and resolve, disruptions. A stockout can now flow all the way back along the supply chain to the literal coal face, stalling the entire business value-chain. Restoring an essential service is delayed while we scramble to procure the vital missing part. Mortgage approvals are deferred while we try reallocate the work load of a valuer dealing with a personal emergency. Or our carefully synchronized product launch falls apart for what seems like a trivial reason somewhere on the other side of the globe.

Our most powerful tools in creating todays high velocity businesses—tools like straight-through processing, LEAN and Six Sigma—worked by removing variation from business processes to increase throughput. The same tools prevent us from effectively responding to these disruptions.

Opportunities today are more frequent, but disruptive and fleeting. An open air festival in the country might represent an opportunity for a tolling operator to manage parking in an adjacent field, if the solution can be deployed as sufficient scale rapidly enough. Or the current trend for pop-up retail stores (if new products rapidly come and go, then why not stores) could be moved from an exceptional, special occasion marketing tool, into the mainstream as a means to optimize sales day-by-day. Responding to these opportunities implies reconfiguring our business on the fly—rapidly integrating business exceptions into the core of our business. This might range from reconfiguring our carefully designed global supply chain, through changing core mortgage approval criteria and processes to modifying category management strategies in (near) real time.

Sam: Waiting while his bank sorts itself out
Sam: Waiting while his bank sorts itself out

We’re entering a time when our ability to change direction, adapting to and leveraging changes in the commercial environment as they occur, will drive our success. If we can react faster than the competition then we can capitalize on a business opportunity and harvest any value the opportunity creates. Our focus will become acceleration: working too build businesses with the flexibility and spare energy required to turn and respond rapidly. These businesses will be the F1 cars of business, providing a massive step in performance over more conventional organizations. And, just like F1, they will also require a new level of performance from our knowledge workers. If acceleration is our focus, then our biggest challenge will be creating time and space required by our knowledge workers to identify these opportunities, turn the steering wheel and leverage them as they occur.

Update: A friend of mine just pointed out that the logical progression of mass → velocity → acceleration naturally leads to jerk, which is an informal unit of measurement for the third derivative.