Tag Archives: data warehouse

Working from the outside in

We’re drowning in a sea of data and ideas, with huge volumes of untapped information available both inside and outside our organization. There is so much information at our disposal that it’s hard to discern Arthur from Martha, let alone optimize the data set we’re using. How can we make sense of the chaos around us? How can we find the useful signals which will drive us to the next level of business performance, from amongst all this noise?

I’ve spent some time recently, thinking about how the decisions our knowledge workers make in planning and managing business exceptions can have a greater impact on our business performance than the logic reified in the applications themselves. And how the quality of information we feed into their decision making processes can have an even bigger impact, as the data’s impact is effectively amplified by the decision making process. Not all data is of equal value and, as is often said, if you put rubbish in then you get rubbish out.

Traditional Business Intelligence (BI) tackles this problem by enabling us to mine for correlations in the data tucked away in our data warehouse. These correlations provide us with signals to help drive better decisions. Managing stock levels based on historical trends (Christmas rush, BBQs in summer …) is good, but connecting these trends to local demographic shifts is better.

Unfortunately this approach is inherently limited. Not matter how powerful your analytical tools, you can only find correlations within and between the data sets you have in the data warehouse, and this is only a small subset of the total data available to us. We can load additional data sets into the warehouse (such as demographic data bought from a research firm), but in a world awash with (potentially useful) data, the real challenge is deciding on which data sets to load, and not in finding the correlations once they are loaded.

What we really need is a tool to help scan across all available data sets and find the data which will provide the best signals to drive the outcome we’re looking for. An outside-in approach, working from the outcome we want to the data we need, rather than an inside-out approach, working from the data we have to the outcomes it might support. This will provide us with a repeatable method, a system, for finding the signals needed to drive us to the next level of performance, rather than the creative, hit-and-miss approach we currently use. Or, in geekier terms, a methodology which enables us to proactively manage our information portfolio and derive the greatest value from it.

I was doodling on the tram the other day, playing with the figure I created for the Inside vs. Outside post, when I had a thought. The figure was created as a heat map showing how the value of information is modulated by time (new vs. old) and distance (inside vs. outside). What if we used it the other way around? (Kind of obvious in hindsight, I know, but these things usually are.) We might use the figure to map from the type of outcome we’re trying to achieve back to the signals required to drive us to that outcome.

Time and distance drive the value of information
Time and distance drive the value of information

This addresses an interesting comment (in email) by a U.K. colleague of mine. (Jon, stand up and be counted.) As Andy Mulholland pointed out, the upper right represents weak confusing signals, while the lower left represents strong, coherent signals. Being a delivery guy, Jon’s first though was how to manage the dangers in excessively focusing on the upper right corner of the figure. Sweeping a plane’s wings forward increases its maneuverability, but at the cost of decreasing it’s stability. Relying too heavily on external, early signals can, in a similar fashion, could push an organization into a danger zone. If we want to use these types of these signals to drive crucial business decisions, then we need to understand the tipping point and balance the risks.

My tram-doodle was a simple thing, converting a heat map to a mud map. For a given business decision, such as planning tomorrow’s stock levels for a FMCG category, we can outline the required performance envelope on the figure. This outline shows us the sort of signals we should be looking for (inside good, outside bad), while the shape of the outlines provides us with an understanding (and way of balancing) the overall maneuverability and stability of the outcome the signals will support. More external predictive scope in the outline (i.e. more area inside the outline in the upper-right quadrant) will provide a more responsive outcome, but at the cost of less stability. Increasing internal scope will provide a more stable outcome, but at the cost of responsiveness. Less stability might translate to more (potentially unnecessary) logistics movements, while more stability would represent missed sales opportunities. (This all creates a little deja vu, with a strong feeling of computing Q values for non-linear control theory back in university, so I’ve started formalizing how to create and measure these outlines, as well as how to determine the relative weights of signals in each area of the map, but that’s another blog post.)

An information performance mud map
An information performance mud map

Given a performance outline we can go spelunking for signals which fit inside the outline.

Luckily the mud map provides us with guidance on where to look. An internal-historical signal is, by definition driven by historical data generated inside the organization. Past till data? An external-reactive signal is, by definition external and reactive. A short term (i.e. tomorrow’s) weather forecast, perhaps? Casting our net as widely as possible, we can gather all the signals which have the potential to drive us toward to the desired outcome.

Next, we balance the information portfolio for this decision, identifying the minimum set of signals required to drive the decision. We can do this by grouping the signals by type (internal-historical, …) and then charting them against cost and value. Cost is the acquisition cost, and might represent a commercial transaction (buying access to another organizations near-term weather forecast), the development and consulting effort required to create the data set (forming your own weather forecasting function), or a combination of the two, heavily influenced by an architectural view of the solution (as Rod outlined). Value is a measure of the potency and quality of the signal, which will be determined by existing BI analytics methodologies.

Plotting value against cost on a new chart creates a handy tool for finding the data sets to use. We want to pick from the lower right – high value but low cost.

An information mud map
An information mud map

It’s interesting to tie this back to the Tesco example. Global warming is making the weather more variable, resulting in unseasonable hot and cold spells. This was, in turn, driving short-term consumer demand in directions not predicted by existing planning models. These changes in demand represented cost, in the from of stock left on the shelves past it’s use-by date, or missed opportunities, by not being able to service the demand when and where it arises.

The solution was to expand the information footprint, pulling in more predictive signals from outside the business: changing the outline on the mud map to improve closed-loop performance. The decision to create an in-house weather bureau represents a straight forward cost-value trade-off in delivering an operational solution.

These two tools provide us with an interesting approach to tackling a number of challenges I’m seeing inside companies today. We’re a lot more externally driven now than we were even just a few years ago. The challenge is to identify customer problems we can solve and tie them back to what our organization does, rather than trying to conceive offerings in isolation and push them out into the market. These tools enable us to sketch the customer challenges (the decisions our customers need to make) and map them back to the portfolio of signals that we can (or might like to) provide to them. It’s outcome-centric, rather than asset-centric, which provides us with more freedom to be creative in how we approach the market, and has the potential to foster a more intimate approach to serving customer demand.

The value of information

We all know that data is valuable; without it it would be somewhat difficult to bill customers and stay in business. Some companies have accumulated masses of data in a data warehouse which they’ve used to drive organizational efficiencies or performance improvements. But do we ever ask ourselves when is the data most valuable?

Billing is important, but if we get the data earlier then we might be able to deal with a problem—a business exception—more efficiently. Resolving a short pick, for example, before the customer notices. Or perhaps even predicting a stock-out. And in the current hyper-competitive business environment where everyone is good, having data and the insight that comes with it just a little bit sooner might be enough to give us an edge.

A good friend of mine often talks about the value of information in a meter. This makes more sense when you know that he’s a utility/energy guru who’s up to his elbows in the U.S. smart metering roll out. Information is a useful thing when you’re putting together systems to manage distributed networks of assets worth billions of dollars. While the data will still be used to drive billing in the end, the sooner we receive the data the more we can do with it.

One of the factors driving the configuration of smart meter networks is the potential uses for the information the meters will generate. A simple approach is to view smart meters as a way to reduce the cost of meter reading; have meters automatically phone readings home rather than drive past each customer’s premisses in a truck and eyeball each meter. We might even used this reduced cost to read the meters more frequently, shrinking our billing cycle, and the revenue outstanding with it. However, the information we’re working from will still be months, or even quarters, old.

If we’re smart (and our meter has the right instrumentation) then we will know exactly which and how many houses have been affected by a fault. Vegetation management (tree trimming) could become proactive by analyzing electrical noise on the power lines that the smart meters can see, and determine where along a power line we need to trim the trees. This lets us go directly to where work needs to be done, rather than driving past every every power line on a schedule—a significant cost and time saving, not to mention an opportunity to engage customers more closely and service them better.

If our information is a bit younger (days or weeks rather than months) then we can use it too schedule just-in-time maintenance. The same meters can watch for power fluctuations coming out of transformers, motors and so on, looking for the tell tail signs of imminent failure. Teams rush out and replace the asset just before it fails, rather than working to a program of scheduled maintenance (maintenance which might be causing some of the failures).

When the information is only minutes old we can consider demand shaping. By turning off hot water heaters and letting them coast we can avoid spinning up more generators.

If we get at or below seconds we can start using the information for load balancing across the network, managing faults and responding to disasters.

I think we, outside the energy industry, are missing a trick. We tend to use a narrow, operational view of the information we can derive from our IT estate. Data is either considered transactional or historical; we’re either using it in an active transaction or we’re using it to generate reports well after the event. We typically don’t consider what other uses we might put the information to if it were available in shorter time frames.

I like to think of information availability in terms of a time continuum, rather than a simple transactional/historical split. The earlier we use the information, the more potential value we can wring from it.

The value of data
The value of data decreases rapidly with age

There’s no end of useful purposes we can turn our information too between the billing and transactional timeframes. Operational excellence and business intelligence allow us to tune business processes to follow monthly or seasonal cycles. Sales and logistics are tuned on a weekly basis to adjust for the dynamics of the current holiday. Days old information would allow us to respond in days, calling a client when we haven’t received their regular order (a non-event). Operations can use hours old information for capacity planning, watching for something trending in the wrong direction and responding before everything falls overs.

If we can use trending data—predicting stock-outs and watching trends in real time—then we can identify opportunities or head off business exceptions before they become exceptional. BAM (business activity monitoring) and real-time data warehouses take on new meaning when viewed in this light.

In a world where we are all good, being smart about the information we can harvest from our business environment (both inside and outside our organization) has the potential to make us exceptional.

Update: Andy Mulholland has a nice build on this idea over at Capgemini‘s CTO blog: Have we really understood what Business Intelligence means?

Why we can’t keep up

We’re struggling to keep up. The pace of business seems to be constantly accelerating. Requirements don’t just slip anymore: they can change completely during the delivery of a solution. And the application we spent the last year nudging over the line into production became instant legacy before we’d even finished. We know intuitively that only a fraction of the benefits written into the business case will be realized. What do we need to do to get back on top of this situation?

We used to operate in a world where applications were delivered on time and on budget. One where the final solution provided a demonstrable competitive advantage to the business. Like SABER, and airline reservation system developed for American Airlines by IBM which was so successful that the rest of the industry was forced to deploy similar solutions (which IBM kindly offered to develop) in response. Or Walmart, who used a data warehouse to drive category leading supply chain excellence, which they leveraged to become the largest retailer in the world. Both of these solutions were billion dollar investments in todays money.

The applications we’ve delivered have revolutionized information distribution both within and between organizations. The wave of data warehouse deployments triggered by Walmart’s success formed the backbone for category management. By providing suppliers with a direct feed from the data warehouse—a view of supply chain state all the way from the factory through to the tills—retailers were able to hand responsibility for transport, shelf-stacking, pricing and even store layout for a product category to their suppliers, resulting in a double digit rises in sales figures.

This ability to rapidly see and act on information has accelerated the pulse of business. What used to take years now takes months. New tools such as Web 2.0 and pervasive mobile communications are starting to convert these months into week.

Take the movie industry for example. Back before the rise of the Internet even bad films could expect a fair run at the box-office, given a star billing and strong PR campaign too attract the punters. However, post Internet, SMS and Twitter, the bad reviews have started flying into punters hands moments after the first screening of a film has started, transmitted directly from the first audience. Where the studios could rely a month or of strong returns, now that run might only last hours.

To compensate, the studios are changing how they take films to market; running more intensive PR campaigns for their lesser offerings, clamping down on leaks, and hoping to make enough money to turn a small profit before word of mouth kicks in. Films are launched, distributed and released to DVD (or even iTunes) in weeks rather than months or years, and studios’ funding, operations and the distribution models are being reconfigured to support the accelerated pace of business.

While the pulse of business has accelerated, enterprise technology’s pulse rate seems to have barely moved. The significant gains we’ve made in technology and methodologies has been traded for the ability to build increasingly complex solutions, the latest being ERP (enterprise resource planning) whose installation in a business is often compared to open heart surgery.

The Diverging Pulse Rates of Business and Technology

This disconnect between the pulse rates of business and enterprise technology is the source of our struggle. John Boyd found his way to the crux of the problem with his work on fighter tactics.

John Boyd—also know as “40 second Boyd”—was a rather interesting bloke. He had a standing bet for 40 dollars that he beat any opponent within 40 seconds in a dog fight. Boyd never lost his bet.

The key to Boyd’s unblemished record was a single insight: that success in rapidly changing environment depends on your ability to orient yourself, decide on, and execute a course of action, faster than the environment (or your competition) is changing. He used his understanding of the current environment—the relative positions, speed and performance envelopes of both planes—to quickly orient himself then select and act on a tactic. By repeatedly taking decisive action faster than his opponent can react, John Boyd’s actions were confusing and unpredictable to his opponent.

We often find ourselves on the back foot, reacting to seemingly chaotic business environment. To overcome this we need to increase the pulse of IT so that we’re operating at a higher pace than the business we support. Tools like LEAN software development have provided us with a partial solution, accelerating the pulse of writing software, but if we want to overcome this challenge then we need to find a new approach to managing IT.

Business, however, doesn’t have a single pulse. Pulse rate varies by industry. It also varies within a business. Back office compliance runs at a slow rate, changing over years as reporting and regulation requirements slowly evolve. Process improvement and operational excellence programs evolve business processes over months or quarters to drive cost out of the business. While customer or knowledge worker facing functionality changes rapidly, possibly even weekly, in response to consumer, marketing or workforce demands.

Aligning technology with business

We can manage each of these pulses separately. Rather than using a single approach to managing technology and treating all business drivers as equals, we can segment the business and select management strategies to match the pulse rate and amplitude of each.

Sales, for example, is often victim of an over zealous CRM (customer relationship management) deployment. In an effort to improve sales performance we’ll decide to role out the latest-greatest CRM solution. The one with the Web 2.0 features and funky cross-sell, up-sell module.

Only of a fraction of the functionality in the new CRM solution is actually new though—the remainder being no different to the existing solution. The need to support 100% of the investment on the benefits provided by a small fraction of the solution’s features dilutes the business case. Soon we find ourselves on the same old roller-coaster ride, with delivery running late,  scope creeping up, the promised benefits becoming more intangible every minute, and we’re struggling to keep up.

There might be an easier way. Take the drugs industry for example. Sales are based on relationships and made via personal calls on doctors. Sales performance is driven by the number of sales calls a representative can manage in a week, and the ability to answer all of a doctor’s questions during a visit (and avoid the need for a follow-up visit to close the sale). It’s not uncommon for tasks unrelated to CRM—simple tasks such as returning to the office to process expenses or find an answer to a question—to consume a disproportionate amount of time. Time that would be better spent closing sales.

One company came up with an interesting approach. To support the sales reps in the field they provided them with the ability to query the team back in the office, answering a clients question without the need to return to head office and then try to get back in their calendar. The solution was to deploy a corporate version of Twitter, connecting the sales rep into the with the call center and all staff using the company portal via a simple text message.

By separating concerns in this way—by managing each appropriately—we can ensure that we are working at a faster pace than the business driver we supporting. By allocating our resources wisely we can set the amplitude of each pulse. Careful management of the cycles will enable us to bring business and technology into alignment.

The Value of Enterprise Architecture

Note: Updated with the slides and script from 2011’s lecture.

Is Enterprise Architecture in danger of becoming irrelevant? And if so, what can we do about it?

Presented as part of RMIT’s Master of Technology (Enterprise Architecture) course.

The Value of Enterprise Architecture

Applications let us differentiate, not!

Being involved in enterprise IT, we tend to think that the applications we build, install and maintain will provide a competitive advantage to the companies we work for.

Take Walmart, for example. During the early 80s, Walmart invested heavily in creating a data warehouse to help it analyze its end-to-end supply chain. The data was used to statically optimize Walmart’s supply chain, creating the most efficient, lowest cost supply chain in the world at the time. Half the savings were passed on to Walmart’s customers, half whet directly to the bottom line, and the rest is history. The IT asset, the data warehouse, enabled Walmart to differentiate, while the investment and time required to develop the data warehouse created a barrier to competition. Unfortunately this approach doesn’t work anymore.

Fast forward to the recent past. The market for enterprise applications has grown tremendously since Walmart first brought that data warehouse online. Today, applications providing solutions to most business problems are available from a range of vendors, and at a fraction of the cost required for the first bespoke solutions that blazed the enterprise application trail. Walmart even replaced that original bespoke supply chain data warehouse, which had become something of an expensive albatross, with an off-the-rack solution. How is it possible for enterprise applications to provide a competitive advantage if we’re all buying from the same vendors?

One argument is that differentiation rests in how we use enterprise applications, rather than in the applications themselves. Think of the manufacturing industries (to use a popular analogy at the moment). If two companies have access to identical factories, then they can still make different, and differentiated, products. Now think of enterprise applications as business process factories. Instead of turning out products, we use these factories to turn out business processes. These digital process factories are very flexible. Even if we all start with the same basic functionality, if I’m smarter at configuring the factory, then I’ll get ahead over time and create a competitive advantage.

This analogy is so general that it’s hard to disagree with. Yes, enterprise applications are (mostly) commodities so any differentiation they might provide now rests in how you use them. However, this is not a simple question of configuration and customization. The problem is a bit more nuanced than that.

Many companies make the mistake that customizing (code changes etc) their unique business processes into an application will provide them with a competitive advantage. Unfortunately the economics of the enterprise software market mean that they are more likely to have created an albatross for their enterprise, than provided a competitive advantage.

Applications are typically parameterized bespoke solutions. (Many of the early enterprise applications were bespoke COBOL solutions where some of the static information—from company name through shop floor configuration—has been pushed into databases as configuration parameters. ) The more configuration parameters provided by the vendor, the more you can bend the application to a shape that suits you.

Each of these configuration parameters requires and investment of time and effort to develop and maintain. They complicate the solution, pushing up its maintenance cost. This leads vendors to try and minimize the number of configuration points they provide to a set of points that will meet most, but not all customers’ needs. In practical terms, it is not possible to configure an application to let you differentiate in a meaningful way. The configuration space is simply too small.

Some companies resort to customizing the application—changing its code—to get their “IP” in. While this might give you a solution reflecting how your business runs today, every customization takes you further from a packaged solution (low cost, easy to maintain, relatively straight forward to upgrade …) and closer to a bespoke solution (high cost, expensive to maintain, difficult or impossible to upgrade). I’ve worked with a number of companies where an application is so heavily customized that it is impossible to deploy vendor patches and/or upgrades. The application that was supposed to help them differentiate had become an expensive burden.

Any advantage to be wrung from enterprise IT now comes from the gaps between applications, not from the applications themselves. Take supply chain for example. Most large businesses have deployed planning and supply chain management solutions, and have been on either the LEAN or Six Sigma journey. Configuring your planning solution slightly differently to your competitors is not going to provide much of an edge, as we’re all using the same algorithms, data models and planning drivers to operate our planning process.

Most of the potential for differentiation now lies with the messier parts of the process, such as exception management (the people who deal with stock-outs and lost or delayed shipments). If I can bring together a work environment that makes my exception managers more productive than yours—responding more rapidly and accurately to exceptions—then I’ve created a competitive advantage as my supply chain is now more agile than yours. If I can capture what it is that my exception managers do, their non-linear and creative problem solving process, automate it, and use this to create time and space for my exception managers to continuously improve how supply chain disruptions are handled, then I’ve created a sustainable competitive advantage. (This is why Enterprise 2.0 is so exciting, since a lot of this IP in this space is tacit information or collaboration.)

Simply configuring an application with today’s best practice—how your company currently does stuff—doesn’t cut it. You need to understand the synergies between your business and the technologies available, and find ways to exploit these synergies. The trick is to understand the 5% that really makes your company different, and then reconfiguring both the business and technology to amplify this advantage while commoditizing the other 95%. Rolls-Royce (appears to be) a great example of getting this right. Starting life as an manufacturer of aircraft engines, Rolls Royce has leveraged its deep understanding of how aircraft engines work (from design through operation and maintenance), reifying this knowledge in a business and IT estate that can provide clients with a service to keep their aircraft moving.