Monthly Archives: May 2010

BPM is not a programming challenge

Get a few beers into a group of developers these days and it’s not uncommon for the complaints to start flowing about BPM (Business Process Management). BPM, they usually conclude, is more pain than it’s worth. I don’t think that BPM is a bad technology, per se, but it does appear to be the wrong tool for the job. The root of the problem is that BPM is a handy tool for programming distributed systems, but the challenge of creating distributed systems is orthogonal to business process execution and management. We’re using a screw driver to belt in a nail. It’s more productive to think business process execution and management as a (realtime) planning problem.

Programming is the automation of the known. Take as stable, repeatable process and automate it; bake the process into silicone to make it go fast. This is the same tactic that I was using back in my image processing days (and that was a long time ago). We’d develop the algorithms in C, experiment and tweak until they were right, and once they were stable we’d burn them into an ASIC (Application-Specific Integrated Circuit) to provide a speed boost. The ASICs were a lot faster than the C version: more than an order of magnitude faster.

Programmers, and IT folk in general, have a habit of treating the problems we confront as programming challenges. This has been outstandingly successful to date; just try and find a home appliance or service that doesn’t have a programme buried in it somewhere. (It’s not an unmitigated success though, such as our tumble drier is driving us nuts if its overly frequent software errors.) It’s not surprising that we chose to treat business processes automation and management as a programming problem once it appeared on our radar.

Don’t get me wrong: BPM is a solid technology. A friend of mine once showed my how he’d used his BPM stack to test its BPEL engine. As side from being a nice example of eating your own dog food, it was a great example of using BPEL as a distributed programming tool to solve a small but complex problem.

So why do we see so many developers complaining about BPM? It’s not the technology itself: the technology works. The issue is that we’re using it to solve problems that it’s not suited for. The most obvious evidence of this is the current poor state of BPM support for business exception management. We’ve deployed a lot of technology to support exception management in business processes without really solving the problem.

Managing business exceptions is driving the developers nuts. I know of one example where managing a couple of not infrequent business exceptions was the major technical problem in a very significant project (well into eight figures). The problem is that business exceptions are not from the same family of beasts as programming exceptions. Programming exceptions are exceptional. Business exceptions are just a (slightly) different way to achieve the same goal. All our compensating actions and exception stacks just get in the way of solving the problem.

On PowerPoint, anything can look achievable. The BPMN diagram we shared with the business was extremely elegant: nice sharp angles and coloured bubbles. Everyone agreed that it was a good representation of what the business does. The devil is in the details though. The development team quickly becomes frustrated as they have to deal with the realities of implementing a dynamic and exception rich business processes. Exceptions pile up on top of exceptions, and soon that BPMN diagram covers a wall, littered as it is with branch and join operations. It’s not a complex process, but we’ve made it incredibly complicated.

Edward Tufte's take on explaining complex concepts with PowerPoint
A military parade explained, a la PowerPoint

We can’t program our way out of this box, trying to pile on more features and patches. We can rip the complications out – simplifying the process to the point that it becomes tractable with our programming tools (which is what happened in my example above). But this removes all the variation which which makes the processes so valuable. (This, of course, the dirty secret of LEAN et al: you’re trading flexibility for cost saving, making your processes very efficient but also very fragile.)

Or we can try solving the problem a different way.

Don’t treat the automation of a business processes as a programming task (and I by this I mean the capture of imperative instructions for a computer to execute, no matter how unstructured or parallel). Programming is the automation of the known. Business processes, however, are the management and anticipation of the unknown. Modelling business processes should be seen as a (realtime) planning problem.

Which comes back to one of my common themes: push vs pull models, or the importance of what over how. Or, as a friend of mine with a better turn of phrase puts it, we need to stop trying to invent new technologies and work out how to use what we already have more effectively. Rather than trying to invent new technologies to solve problems that are already well understood elsewhere, pushing the technology into the problem, a more pragmatic approach is to leverage that existing understanding and then pull in existing technologies as appropriate.

Planning and executing in a rapidly changing environment is a well understood problem. Just ask anyone who’s been involved with the military. If we view the management of a business processes as a realtime planning problem, then what were business exceptions are reduced to simply alternate routes to the same goal, rather than a problem which requires a compensating action.

Battle of Gaugamela (Arbela) (331BC)
Take that hill!

One key principle is to establish a clear goal – Take that hill!, or Find that lost shipment! – articulate the tactics, the courses of action we might use to achieve that goal, and then defer decisions on which course of action to take until the decision needs to be made. If we commit to a course of action too early, locking in a decision during design time, then it’s likely that we’ll be forced to manage the exception when we realise that we picked the wrong course of action. It’s better to wait until the moment when all relevant information and options are available to us, and then take decisive action.

From a modelling point of view, we need to establish where are the key events at which we need to make decisions in line with a larger strategy. The decisions at each of these events needs to weigh the available courses of action and select the most appropriate, much like using a set of business rules to identify applicable options. The course of action selected, a scenario or business process fragment, will be semi independent from the other in the applicable set, as it addresses a different business context. Nor can the scenario we pick cannot be predetermined, as it depends on the business context. Short and sharp, each scenario will be simple, general and flexible, enabling us to configure it for the specific circumstances at hand, as we can’t anticipate all possible scenarios. And finally, we need to ensure that the scenarios we provide cover the situations we can anticipate, including the provision of a manual escape hatch.

Goals, rules and process: in that order. Integrated rather than as standalone engines. Pull pull these established technologies into a single platform and we might just be closer to a BPM solution inline with what we really need. (And we know there is nothing new under the sun, as this essentially a build on Jim Sinurs rules-and-process argument, and borrows a lot from STRIPS, PRS, dMARS and even the work I did at Agentis.)

As I mentioned at the start of this missive, BPM as a product category makes sense and the current implementations are capable distributed programming tools. The problem is that business process management is not a distributed programming challenge. Business exceptions are not exceptional. I say steal a page from the military strategy book – they, after all, have been successfully working on this problem for some time – and build our solutions around ideas the military use to succeed in a rapidly changing environment. Goals, rules and processes. The trick is to be pragmatic, rather than dogmatic in our implementation, and focus on solving the problem rather then trying to create a new technology.

Innovation [2010-05-25]

Another week and another collection of interesting ideas from around the internet.

As always, thoughts and/or comments are greatly appreciated.

  • Innovation and R&D [The Economist]
    The Economist provides two interactive visualisations that show the future of innovation taking shape. We can see a strong link between the number of international patents that a country is granted and the amount that it spends on research and development. A 2007 snapshot shows this clearly, and also that America and Japan led the pack.
  • McLaren wins the innovation race [timkastelle.org]
    McLaren shows us how to leverage lessons learnt in Formula 1 in other domains, such as air traffic control and health care, making them the masters of extending current competencies into new fields.
  • The creative process gone wrong [Bob Sutton]
    It seems that the very tools we often use to manage the creative, innovation process can be the source of the process’s biggest problems.
  • What if the very theory that underlies why we need patents is wrong? [techdirt]
    Our current patent system was designed for an earlier, gentler age. A number of pundits are – quite rightly – pointing out that the system is tremendously obsolete in terms of actually promoting the progress, and is set up in a way that favors a concept of innovation and invention that may not be how the world actually works today.

Danger Will Robinson!

Ack! The scorecard's gone red!
Ack! The scorecard's gone red!

Andy Mulholland has a nice post over at the Capgemini CTO blog, which points out that we have a strange aversion to the colour red. Having red on your balanced scorecard is not necessarily a bad thing, as it tells you something that you didn’t know before. Insisting on managers delivering completely green scorecard is just throwing good information away.

Unfortunately something’s wrong with Capgemini’s blogging platform, and it won’t let me post a comment. Go and read the post, and then you can find my comment below.

Economists have a (rather old) saying: “if you don’t fail occasionally, then you’re not optimising (enough)”. We need to consider red squares on the board to be opportunities, just as much as they might be problems. Red just represents “something happened that we didn’t expect”. This might be bad (something broke), or it might be good (an opportunity).

Given the rapid pace of change today, and the high incidence of the unexpected, managing all the red out of your business instantly turns you into a dinosaur.

People don’t like change. (Or do they?)

I seem to be having a lot of conversations at the moment around whether people (you, me and everyone else) like and embrace change, or whether they resist it. The same question arises for companies. Like a lot of these questions, I think it depends. As individuals we don’t mind change, given appropriate circumstances. Organisations also want to change (and in today’s business environment it seems to be a question of changing or becoming irrelevant). However, people in organisations are usually strongly incentivised to dislike change, especially if they want to make that next repayment on their new mortgage. Fixing this, and creating a culture that embraces change, means changing the way we think about and structure our organisations and our careers. It means rethinking the rules of enterprise IT.

Every time a conversation comes around to the topic of change, I’m always reminded of a visit I made to a Toyota factory something like a decade ago. It’s so long ago now that I can’t even remember the reason for the visit, but that’s not the point of this missive.

Toyota, like most businesses, loves change. (Many large companies reorganise so often that change seems to be the only constant.) Change, embodied in the development of the Toyota Production System, was what took Toyota from the bottom of the global car industry to the top. Change is also why many of us have moved companies, following jobs as our employers reorganise their operations. For some of us, change is an opportunity. For many though, change is the tool of the man as he tries to disrupt our lives. Change means unwanted relocations, pay cuts, career stalls, or the need to shift jobs when we don’t want to. Change is something to be resisted.

What was interesting about my visit to Toyota though, was the attitude of the workers on the shop floor had to change. They didn’t hate it. They didn’t even resist it. They actually arrived at work each day eager to see how work practices had been changed since the end of their last shift.

A Toyota assembly line circa 2000.
A Toyota assembly line circa 2000

The concrete example I saw of this was the pre-sorting of seatbelt parts into coloured tubs. Apparently only a few weeks earlier the parts had been arranged on a wall. All hooks and dangling parts, like your Dad’s tools in the shed. When a car came down the assembly line a worker would select the parts appropriate for the car model, and then attach them to the car. Each seat belt had roughly four parts, so that meant there was three unnecessary decisions. Unnecessary decisions usually mean mistakes, mistakes waste time and money, and there were a number of mistakes made.

One day a member of the shop floor team had had the bright idea of pre-sorting the seat belts to avoid these mistakes. Some coloured tubs were sourced (some of the shop floor team drove to the local Walmart with a little petty cash), parts were sorted into tubs, and they gave the idea a trial run. Selecting seatbelt parts for a car now only required one decision: which tub?

The idea was a huge success; error rates went down dramatically. I hear that it was even taken global, and implemented in most Toyota factories around the world. (Though being around ten years ago, and with today’s rapid pace of change, I expect that the tubs have been superseded by now.)

What’s interesting about this story is that the change originated on the shop floor, from the assembly line worker who were actively looking to improve operations, rather than from head office as part of a reorganisation. Some of the improvements I heard about even resulted in the elimination of jobs, with the workers redeployed elsewhere in the factory. Workers weren’t just changing how they did something, they were also changing what they did. Change was what made the work interesting and engaging for the workers, rather than being seen as an oppressive tool used by the man.

I think we can safely set aside the idea that works don’t like change, as this story is not an isolated incident. Why then, do so many people resist change? Why, for every Toyota factory, there is a story like the UK newspaper industry, where workers (and unions) resisted change for decades, until Rupert Murdoch came along.

Rupert Murdoch, destroyer of unions, and good Melbourne boy
Rupert Murdoch, destroyer of unions, and good Melbourne boy

The problem is not people or organisations, but people in organisations.

People are funny things; they tend to do what you incentivise them to do. There’s an excellent article over at the NY Times, The no-stars all-star, which talks about measurement and incentives in basketball. We often talk about “what gets measured determines what gets done” from an employee incentive point of view, but this article puts some real meat on the bones of that argument.

Shane Battier, the no-stars all-star
Shane Battier, the no-stars all-star

As the article says (on page two):

There is a tension, peculiar to basketball, between the interests of the team and the interests of the individual. The game continually tempts the people who play it to do things that are not in the interest of the group.

A little later it goes on to mention (on page three):

A point guard might selfishly give up an open shot for an assist. You can see it happen every night, when he’s racing down court for an open layup, and instead of taking it, he passes it back to a trailing teammate. The teammate usually finishes with some sensational dunk, but the likelihood of scoring nevertheless declined. “The marginal assist is worth more money to the point guard than the marginal point,” Morey says.

The point guard’s career is defined by the number of assists he makes (among other metrics), and he’ll try and increase the number of assets even if it’s not in the best interest of the team. After all, teams come and go, while he has a career to maintain.

Once you place a person into a role you have put them on a career path which will determine their attitude to change.

Usually we take an operational approach to defining roles, rewarding people for the volume of work they are responsible for. Career progression then means increasing the amount of work they are responsible for, regardless of what this means for the company.

Measuring a project manager in terms of head count or revenue under management will give them a strong preference for creating ever bigger projects. It doesn’t matter if the right thing to do is create more, smaller projects, rather than run a programme of a few major projects as we have in the past. Your project manager’s career path is to increase their head count and revenue under management. And they do have those private school fees due soon.

Just like the point guard, change that will prevent career progression will be resisted (remember those kids in private school), even if it is counter to the company’s best interests. Which makes the current transformation we’re seeing in IT all the more important, because if we set the wrong incentives in place then we just might be our own worst enemies.

We can’t force a square peg into a round hole; nor can we force our existing employees to take their current roles and careers into a new organisational model. They just don’t fit. Take IT for example. We can’t expect many modern IT departments to spontaneously modernise themselves, transforming into agile business-technology engines under their own volition. It’s not that the departments don’t want to change: they do. Nor are most of the employees, as individuals, opposed (remember the Toyota example). But the combination of people and organisation will repel all but the most destructive boarders.

It’s interesting how other games, games other than basketball that is, have structural solutions to this problem. One solution is the line-up in baseball. From the NY Times article (page two, again):

“There is no way to selfishly get across home plate,” as Morey puts it. “If instead of there being a lineup, I could muscle my way to the plate and hit every single time and damage the efficiency of the team — that would be the analogy.”

Solving this problem in IT means rethinking the rules of IT.

The game of IT has, for the last few decades, been determined by the need to deliver large, enterprise applications into the IT estate. Keep the lights on, don’t lose orders, and automate anything that hasn’t yet been automated. Oh — and I’d like my reports accurate and on time. IT as the game of operational engineering. It was these rules that drove the creation of most of the roles we have in enterprise IT today.

However, this has changed. Decisions are now more important than data, and the global credit crunch is driving us to reconsider the roles we need in IT. We’re trying to reinvent our IT departments for the modern era – I even posted about how this was driving the need the need to manage technology, and not applications – but we haven’t changed the rules to suit.

If we want out people to embrace change, as the people on the shop floor at Toyota did, then we need to provide them with roles and careers that support them in the new normal. And this means changing the rules. Out with the more – more applications, larger projects, more people – and in with the new.

So what are the new rules for IT?

Tea bags: the unexpected

As seen on a plaque at Scienceworks in the House Secrets exhibit.

A thrifty tea merchant from New York named Thomas Sullivan is credited with inventing the first tea bag in 1908. Looking to save money, Sullivan reportedly distributed small samples of tea in silk bags instead of little metal tins. It wasn’t until after he saw restaurant and coffee shop owners brewing the entire bag of tea leaves that he realized the potential of his actions.

A nice demonstration of the first, and most valuable, of Peter Drucker’s seven sources of innovation.

The unexpected. The unexpected success, failure or outside event.

Posted via web from PEG @ Posterous

Decisions are more important than data

Names and categories are important. Just look at the challenges faced by the archeology community as DNA evidence forces history to be rewritten when it breaks old understandings, changing how we think and feel in the process. Just who invaded who? Or was related to who?

We have the same problem with (enterprise) technology; how we think about the building blocks of the IT estate has a strong influence on how approach the problems we need to solve. Unfortunately our current taxonomy has a very functional basis, rooted as it is in the original challenge of creating the major IT assets we have today. This is a problem, as it’s preventing us to taking full advantage of the technologies available to us. If we want to move forward, creating solutions that will thrive in a post GFC world, then we need to think about enterprise IT in a different way.

Enterprise applications – the applications we often know and love (or hate) – fall into a few distinct types. A taxonomy, if you will. This taxonomy has a very functional basis, founded as it is on the challenge of delivering high performance and stable solutions into difficult operational environments. Categories tend to be focused on the technical role a group of assets have in the overall IT estate. We might quibble over the precise number of categories and their makeup, but for the purposes of this argument I’m going to go with three distinct categories (plus another one).

SABER
SABER @ American Airlines

First, there’s the applications responsible for data storage and coherence: the electronic filing cabinets that replaced rooms full of clerks and accountants back in the day. From the first computerised general ledger through to CRM, their business case is a simple one of automating paper shuffling. Put the data in on place and making access quick and easy; like SABER did, which I’ve mentioned before.

Next, are the data transformation tools. Applications which take a bunch of inputs and generate an answer. This might be a plan (production plan, staffing roster, transport planning or supply chain movements …) or a figure (price, tax, overnight interest calculation). State might be stored somewhere else, but these solutions still need some some serious computing power to cope with hugh bursts in demand.

Third is data presentation: taking corporate information and presenting in some form that humans can consume (though looking at my latest phone bill, there’s no attempt to make the data easy to consume). This might be billing or invoicing engines, application specific GUIs, or even portals.

We can also typically add one more category – data integration – though this is mainly the domain of data warehouses. Solutions that pull together data from multiple sources to create a summary view. This category of solutions wouldn’t exist aside from the fact that our operational, data management solutions, can’t cope with an additional reporting load. This is also the category for all those XLS spreadsheets that spread through business like a virus, as high integration costs or more important projects prevent us from supporting user requests.

A long time ago we’d bake all these layers into the one solution. SABER, I’m sure, did a bit of everything, though its main focus was data management. Client-server changed things a bit by breaking user interface from back-end data management, and then portals took this a step further. Planning tools (and other data transformation tools) started as modules in larger applications, eventually popping out as stand alone solutions when they grew large enough (and complex enough) to justify their own delivery effort. Now we have separate solutions in each of these categories, and a major integration problem.

This categorisation creates a number of problems for me. First and foremost is the disconnection between what business has become, and what technology is trying to be. Back in the day when “computer” referred to someone sitting at a desk computing ballistics tables, we organised data processing in much the same way that Henry Ford organised his production line. Our current approach to technology is simply the latest step in the automation of this production line.

Computers in the past
Computers in the past

Quite a bit has changed since then. We’ve reconfigured out businesses, we’re reconfiguring our IT departments, and we need to reconfigure our approach to IT. Business today is really a network of actors who collaborate to make decisions, with most (if not all) of the heavy data lifting done by technology. Retail chains are trying to reduce the transaction load on their team working the tills so that they can focus on customer relationships. The focus in supply chains to on ensuring that your network of exception managers can work together to effectively manage disruptions in the supply chain. Even head office focused on understanding and responding to market changes, rather than trying to optimise the business in an unchanging market.

The moving parts of business have changed. Henry Ford focused on mass: the challenge of scaling manufacturing processes to get cost down. We’re moved well beyond mass, through velocity, to focus on agility. A modern business is a collection of actors collaborating and making decisions, not a set of statically defined processes backed by technology assets. Trying to force modern business practices into yesterdays IT taxonomy is the source of one of the disconnects between business and IT that we complain so much about.

There’s no finer example of this than Sales and Operations Planning (S&OP). What should be a collaborative and fluid process – forward planning among a network of stakeholders – has been shoehorned into a traditional n-tier, database driven, enterprise solution. While an S&OP solution can provided significant cost saving, many companies find it too hard to fit themselves into the solution. It’s not surprising that S&OP has a reputation for being difficult to deploy and use, with many planners preferring to work around the system than with it.

I’ve been toying with a new taxonomy for a little while now, one that tries to reflect the decision, actor and collaboration centric nature of modern business. Rather than fit the people to the factory, which was the approach during the industrial revolution, the idea is to fit the factory to the people, which is the approach we use today post LEAN and flexible manufacturing. While it’s a work in progress, it still provides a good starting point for discussions on how we might use technology to support business in the new normal.

In no particular order…

Fusion solutions blend data and process to create a clear and coherent environment to support specific roles and decisions. The idea is to provide the right data and process, at the right time, in a format that is easy to consume and use, to drive the best possible decisions. This might involve blending internal data with externally sourced data (potentially scraped from a competitor’s web site); whatever data required. Providing a clear and consistent knowledge work environment, rather than the siloed and portaled environment we have today, will improve productivity (more time on work that matters, and less time on busy work) and efficiency (fewer mistakes).

Next, decisioning solutions automate key decisions in the enterprise. These decisions might range from mortgage approvals through office work, such as logistics exception management, to supporting knowledge workers workers in the field. We also need to acknowledge that decisions are often decision making processes which require logic (roles) applied over a number of discrete steps (processes). This should not be seen as replacing knowledge workers, as a more productive approach is to view decision automation as a way of amplifying our users talents.

While we have a lot of information, some information will need to be manufactured ourselves. This might range from simple charts generated from tabular data, through to logistics plans or maintenance scheduling, or even payroll.

Information and process access provide stakeholders (both people and organisations) with access to our corporate services. This is not your traditional portal to web based GUI, as the focus will be on providing stakeholders with access wherever and whenever they need, on whatever device they happen to be using. This would mean embedding your content into a Facebook app, rather than investing in a strategic portal infrastructure project. Or it might involve developing a payment gateway.

Finally we have asset management, responsible for managing your data as a corporate asset. This looks beyond the traditional storage and consistency requires for existing enterprise applications to include the political dimension, accessibility (I can get at my data whenever and wherever I want to) and stability (earthquakes, disaster recovery and the like).

It’s interesting to consider the sort of strategy a company might use around each of these categories. Manufacturing solutions – such as crew scheduling – are very transactional. Old data out, new data in. This makes them easily outsourced, or run as a bureau service. Asset management solutions map very well to SaaS: commoditized, simple and cost effective. Access solutions are similar to asset management.

Fusion and decisioning solutions are interesting. The complete solution is difficult to outsource. For many fusion solutions, the data and process set presented to knowledge workers will be unique and will change frequently, while decisioning solutions contain decisions which can represent our competitive advantage. On the other hand, it’s the intellectual content in these solutions, and not the platform, which makes them special. We could sell our platform to our competitors, or even use a commonly available SaaS platform, and still retain our competitive advantage, as the advantage is in the content, while our barrier to competition is the effort required to recreate the content.

This set of categories seems to map better to where we’re going with enterprise IT at the moment. Consider the S&OP solution I mention before. Rather than construct a large, traditional, data-centric enterprise application and change our work practices to suit, we break the problem into a number of mid-sized components and focus on driving the right decisions: fusion, decisioning, manufacturing, access, and asset management. Our solution strategy becomes more nuanced, as our goal is to blend components from each category to provide planners with the right information at the right time to enable them to make the best possible decision.

After all, when the focus is on business agility, and when we’re drowning in a see of information, decisions are more important than data.

Complexity isn’t (at least in enterprise IT)

There’s a lot of talk these days about complexity in enterprise IT. The heterogeneous solutions we’re building today seem more complex than the monolithic solutions of the past. But are they really? I’ll grant you that a lot of what is being built at the moment is complicated. Complex though? I don’t think so. The problem is that we’re building new things in old ways when we need to be building new things in new ways.

I’ve always used a simple rule of thumb when thinking about complexity. Some folk like to get fancy with two and three dimensional models that enable us to ascribe degrees of complexity to problems. While I find these models interesting, my focus has always been on how do I solve the problem in front of me. What is the insight that will make the hard easy? For me, one simple distinction seems to provide the information I need. Is a solution complex? Or is it complicated?

Something is complicated if the model we use to understand the problem requires patches, exceptions to make it work. The model might be simple and well understood, but we’re forced to patch the model for it to succeed when confronted by the real world; we’re adding epicycles. It’s not a complex system, but it is complicated.

Adding epicycles didn't manage to keep the earth at the centre of the universe
Adding epicycles didn't manage to keep the earth at the centre of the universe

On the other hand, something is complex if it’s difficult to develop a consistent model for the problem. While we might have a well understood model, it’s definitely not simple, requiring a great deal of academic and tacit knowledge to understand. There’s no epicycles, but there are a large number of variables involved, and their interactions are often non-linear.

While this binary separation might not be strictly true (the complicated can sometimes be complex), I find that the truly complex problems are rare enough that the rule of thumb is useful most of the time. After all, that’s what a rule of thumb is. The few times that it breaks down, experience comes to the rescue.

Distinguishing between the complex and the complicated is not hard; just look for the epicycles. Planning engines – such as material planning or crew scheduling – are a good example of a complex solutions. Business processes management is a good example of a complicated solution. Psi calculus – the model at the heart of a modern BPM engine – is well understood and BPM engines work as described. However, managing business exceptions is a mess as support for them is tacked on rather than an inherent part of the model. Smashing together psi calculus, transactions and number of other models has resulted in epicycles.

Most of the problems we’re seeing in enterprise IT are complicated, but not complex. Take the current efforts to create IT estates integrating SaaS and public cloud with more traditional enterprise IT tools, such as on-premesis applications and BPO. Conventional approaches to understanding and planning IT estates are creaking at the seems. The model – the enterprise integration patterns – which we’ve used for so long is well understood, but it’s creaking at the seams as we bolt on more epicycles to cope with exceptions as they arise.

Dr. Khee Pang
Dr. Khee Pang

A great piece of advice from a former lecturer of mine always comes to mind in situations like this. As I’ve mentioned before:

If you don’t like a problem, then change it.
KK Pang

Our solution is complicated because we’re trying to solve the wrong problem. We need to change it.

The problem with BPM is that business exceptions are not exceptional, but are simply alternative ways of achieving the same goals. To resolve the epicycles we need to shift the problems centre of gravity, moving the earth from the centre of the universe to a more stable orbit. If business exceptions are not exceptional, then we should simply consider them as different business scenarios, and use a scenario based approach to capturing business processes. The epicycles then melt away.

I think we can use a similar approach to help us with the challenges we’re seeing in today’s IT estates, the same challenges which are trigger some of the discussion on complexity. The current approach to planning and provisioning IT is data centric; most applications are, after all, just large data management engines. This data-centric approach is forcing us to create epicycles as we attempt to integrate SaaS, cloud, and a whole raft of new tools and techniques. The solution is to move from a data-centric approach, to decision-centric approach. But that’s a different blog post.

Innovation [2010-05-10]

Another week and another collection of interesting ideas from around the internet.

As always, thoughts and/or comments are greatly appreciated.

Penicillin: the unexpected

As seen on a plaque at Scienceworks.

The penicillin mold was a pest, not a resource. Backteriologists went to great lengths to protect their bacterial cultures against contamination by it. Then in the 1920s, a London doctor, Alexander Fleming, realized that this “pest” was exactly the bacterial killer bacteriologists had been looking for – and the penicillin mold became a valuable resource.

A nice demonstration of the first, and most valuable, of Peter Drucker‘s seven sources of innovation.

The unexpected. The unexpected success, failure or outside event.

Posted via web from PEG @ Posterous

Having too much SOA is a bad thing (and what we might do about it)

SOA enablement projects (like a lot of IT projects) have a bad name. An initiative that starts as a good idea to create a bit more flexibility in the IT estate often seems to end up mired in its own complexity. The problem is usually too much flexibility, as flexibility creates complexity, and complexity exponentially increases the effort required to manage and deliver the software. Without any solid guidance on how much flexibility to create (and where to create it) most SOA initiatives simply keep creating flexibility until either the project collapses under its own weight, or the projected development work to create all the services exceeds the available CAPEX budget. A little flexility is good, but too much is bad. How can we scope the flexibility, pointing it where it’s most needed while preventing it from becoming a burden?

The challenge with SOA enablement is in determining how much flexibility to build into the IT estate. Some flexibility is good – especially if it’s focused on where the business needs it the most – but too much flexibility is simply another unnecessary cost. The last decade or so is littered with stories of companies who’s SOA initiatives were either brought to an early close or canned as they had consumed all the cash the business was prepared to invest into a major infrastructure project. Finance and telecoms seem particularly prone of creating these gold-plated SOA initiatives. (How many shelf-ware SDFs – service delivery frameworks – do you know of?)

The problem seems to be a lack of guidance on how much flexibility to build, or where to put it. We sold the business on the idea that a flexible, service-oriented IT estate would be better then the evil monolithic applications of old, but the details of just how flexible the new estate would be were a little fuzzy. Surely these details can be sorted out in service discovery? And governance should keep service discovery on track! We set ourselves up by over-promising and under-delivering.

Mario Batali: Too much is never enough!
Mario Batali

This much was clear: the business wanted agility, and agility requires flexibility. As flexibility comes from having more moving parts (services), we figured that creating more moving parts will create more agility. Service discovery rapidly became a process of identifying every bit of (reusable) functionality that we can pack into a service. More is better, or, as the man with the loud shoes says:

Too much is never enough!
Mario Batali

The problem with this approach is that it confuses flexibility and agility. It’s possible to be very flexible without being agile, and vica versa. Think of a formula one car: they’re fast and they’re agile (which is why driving them tends to be a young mans game), and they’re very stiff. Agility comes from keeping the weight down and being prepared to act quickly. This means keeping things simple, ensuring that we have minimum set of moving parts required. They might have an eye for detail, such as nitrogen in the tyres, but unnecessary moving parts that might reduce reliability or performance are eliminated.

This gold plated approach to SOA creates a lot of unrequired flexibility, this additional flexibility increases complexity, and the complexity becomes the boat anchor that slows you down and stops you from being agile. Turning the car is no longer a simple of tugging on the steering wheel, as we need governance to stop us from pulling the wrong lever in the bank of 500 identical levers in front of us.

It's really that simple!
It's really that simple!

We’ve made everything too complicated. Mario was wrong: too much is too much.

What we need is some guidance – a way of scoping and directing the flexibility we’re going to create. Governance isn’t enough, as governance is focused on stopping bad things from happening. We have a scoping problem. Our challenge is to understand what flexibility will be required in the future, and agreeing on the best way to support it.

To date I’ve been using a very fuzzy “business interest” metric for this, where services are decomposed until the business is no longer interested. The rational is that we put the flexibility only were the business thinks it needs to focus. This approach works fairly well, but it relies too much on the tacit judgement of a few skilled business analysts and architects, making it too opaque and hard to understand for the people not involved in the decision making process. It’s also hard to scale. We need something more deterministic and repeatable.

Which brings me to a friend’s MBA thesis, which he passed to me the other week. It’s an interesting approach to building business cases for IT solutions, one based on real options.

The problem with the usual approaches to building a business case, using tools like net present value (NPV) and discounted cash flow, is that we assume that the world doesn’t change post the decision to build the solution (or not). They don’t factor in the need to change a solution once it’s in the field, or even during development.

The world doesn’t work this way: the solution you approved in yesterday’s business environment will be deployed into a radically different business environment tomorrow. This makes it hard to justify the additional investment required for a more flexible SOA based solution, when compared to a conventional monolithic solution. The business case doesn’t include flexibility as a factor, so more flexible (and therefore complex and expensive) solutions lose to the cheaper, monolithic approach.

Real options address this by pushing you down a scenario planning based approach. You estimate the future events that you want to guard against, and their probabilities, creating a set of possible futures. Each event presents you with options to take action. The action, for example, might be to change, update or replace components in the solution to bring them in line with evolving business realities. The options are – in effect – flex-points that we might design into our solutions SOA. The real options methodology enables us to ascribe costs to these future events and the create a decision tree that captures the benefits of investing in specific flex points, all in a clear and easily understandable chain of reasoning.

The decision tree and options provide us with a way to map out where to place flex points in the SOA solution. They also provide us with strong guidance on how much flexibility to introduce. And this is the part I found really interesting about the approach. It also provides us with a nice framework to govern the evolution of the SOA solution, as changes are (generally) only made when an option is taken: when it’s business case is triggered.

It’s a bit like those formula one cars. A friend of mine used to work for one F1 manufacturer designing and testing camshafts. These camshafts had to fall within a 100,000 lifetime revolution window. An over-designed camshaft was unnecessary weight, while an under-designed one means that you wouldn’t win (or possibly even finish) the race. Work it out: a 100,000 revolutions is a tiny window for an F1 car, given the length of a race.

An approach like real options helps us ensure that we only have the flexibility required in the solution, and that it is exactly where it is required. Not too much, and not too little. Just enough to help us win the race.