The value of information

We all know that data is valuable; without it it would be somewhat difficult to bill customers and stay in business. Some companies have accumulated masses of data in a data warehouse which they’ve used to drive organizational efficiencies or performance improvements. But do we ever ask ourselves when is the data most valuable?

Billing is important, but if we get the data earlier then we might be able to deal with a problem—a business exception—more efficiently. Resolving a short pick, for example, before the customer notices. Or perhaps even predicting a stock-out. And in the current hyper-competitive business environment where everyone is good, having data and the insight that comes with it just a little bit sooner might be enough to give us an edge.

A good friend of mine often talks about the value of information in a meter. This makes more sense when you know that he’s a utility/energy guru who’s up to his elbows in the U.S. smart metering roll out. Information is a useful thing when you’re putting together systems to manage distributed networks of assets worth billions of dollars. While the data will still be used to drive billing in the end, the sooner we receive the data the more we can do with it.

One of the factors driving the configuration of smart meter networks is the potential uses for the information the meters will generate. A simple approach is to view smart meters as a way to reduce the cost of meter reading; have meters automatically phone readings home rather than drive past each customer’s premisses in a truck and eyeball each meter. We might even used this reduced cost to read the meters more frequently, shrinking our billing cycle, and the revenue outstanding with it. However, the information we’re working from will still be months, or even quarters, old.

If we’re smart (and our meter has the right instrumentation) then we will know exactly which and how many houses have been affected by a fault. Vegetation management (tree trimming) could become proactive by analyzing electrical noise on the power lines that the smart meters can see, and determine where along a power line we need to trim the trees. This lets us go directly to where work needs to be done, rather than driving past every every power line on a schedule—a significant cost and time saving, not to mention an opportunity to engage customers more closely and service them better.

If our information is a bit younger (days or weeks rather than months) then we can use it too schedule just-in-time maintenance. The same meters can watch for power fluctuations coming out of transformers, motors and so on, looking for the tell tail signs of imminent failure. Teams rush out and replace the asset just before it fails, rather than working to a program of scheduled maintenance (maintenance which might be causing some of the failures).

When the information is only minutes old we can consider demand shaping. By turning off hot water heaters and letting them coast we can avoid spinning up more generators.

If we get at or below seconds we can start using the information for load balancing across the network, managing faults and responding to disasters.

I think we, outside the energy industry, are missing a trick. We tend to use a narrow, operational view of the information we can derive from our IT estate. Data is either considered transactional or historical; we’re either using it in an active transaction or we’re using it to generate reports well after the event. We typically don’t consider what other uses we might put the information to if it were available in shorter time frames.

I like to think of information availability in terms of a time continuum, rather than a simple transactional/historical split. The earlier we use the information, the more potential value we can wring from it.

The value of data
The value of data decreases rapidly with age

There’s no end of useful purposes we can turn our information too between the billing and transactional timeframes. Operational excellence and business intelligence allow us to tune business processes to follow monthly or seasonal cycles. Sales and logistics are tuned on a weekly basis to adjust for the dynamics of the current holiday. Days old information would allow us to respond in days, calling a client when we haven’t received their regular order (a non-event). Operations can use hours old information for capacity planning, watching for something trending in the wrong direction and responding before everything falls overs.

If we can use trending data—predicting stock-outs and watching trends in real time—then we can identify opportunities or head off business exceptions before they become exceptional. BAM (business activity monitoring) and real-time data warehouses take on new meaning when viewed in this light.

In a world where we are all good, being smart about the information we can harvest from our business environment (both inside and outside our organization) has the potential to make us exceptional.

Update: Andy Mulholland has a nice build on this idea over at Capgemini‘s CTO blog: Have we really understood what Business Intelligence means?

Innovation [2009-06-29]

Another week and another collection of interesting ideas from around the internet.

As always, thoughts and/or comments are greatly appreciated.

This issue:

Why we can’t keep up

We’re struggling to keep up. The pace of business seems to be constantly accelerating. Requirements don’t just slip anymore: they can change completely during the delivery of a solution. And the application we spent the last year nudging over the line into production became instant legacy before we’d even finished. We know intuitively that only a fraction of the benefits written into the business case will be realized. What do we need to do to get back on top of this situation?

We used to operate in a world where applications were delivered on time and on budget. One where the final solution provided a demonstrable competitive advantage to the business. Like SABER, and airline reservation system developed for American Airlines by IBM which was so successful that the rest of the industry was forced to deploy similar solutions (which IBM kindly offered to develop) in response. Or Walmart, who used a data warehouse to drive category leading supply chain excellence, which they leveraged to become the largest retailer in the world. Both of these solutions were billion dollar investments in todays money.

The applications we’ve delivered have revolutionized information distribution both within and between organizations. The wave of data warehouse deployments triggered by Walmart’s success formed the backbone for category management. By providing suppliers with a direct feed from the data warehouse—a view of supply chain state all the way from the factory through to the tills—retailers were able to hand responsibility for transport, shelf-stacking, pricing and even store layout for a product category to their suppliers, resulting in a double digit rises in sales figures.

This ability to rapidly see and act on information has accelerated the pulse of business. What used to take years now takes months. New tools such as Web 2.0 and pervasive mobile communications are starting to convert these months into week.

Take the movie industry for example. Back before the rise of the Internet even bad films could expect a fair run at the box-office, given a star billing and strong PR campaign too attract the punters. However, post Internet, SMS and Twitter, the bad reviews have started flying into punters hands moments after the first screening of a film has started, transmitted directly from the first audience. Where the studios could rely a month or of strong returns, now that run might only last hours.

To compensate, the studios are changing how they take films to market; running more intensive PR campaigns for their lesser offerings, clamping down on leaks, and hoping to make enough money to turn a small profit before word of mouth kicks in. Films are launched, distributed and released to DVD (or even iTunes) in weeks rather than months or years, and studios’ funding, operations and the distribution models are being reconfigured to support the accelerated pace of business.

While the pulse of business has accelerated, enterprise technology’s pulse rate seems to have barely moved. The significant gains we’ve made in technology and methodologies has been traded for the ability to build increasingly complex solutions, the latest being ERP (enterprise resource planning) whose installation in a business is often compared to open heart surgery.

The Diverging Pulse Rates of Business and Technology

This disconnect between the pulse rates of business and enterprise technology is the source of our struggle. John Boyd found his way to the crux of the problem with his work on fighter tactics.

John Boyd—also know as “40 second Boyd”—was a rather interesting bloke. He had a standing bet for 40 dollars that he beat any opponent within 40 seconds in a dog fight. Boyd never lost his bet.

The key to Boyd’s unblemished record was a single insight: that success in rapidly changing environment depends on your ability to orient yourself, decide on, and execute a course of action, faster than the environment (or your competition) is changing. He used his understanding of the current environment—the relative positions, speed and performance envelopes of both planes—to quickly orient himself then select and act on a tactic. By repeatedly taking decisive action faster than his opponent can react, John Boyd’s actions were confusing and unpredictable to his opponent.

We often find ourselves on the back foot, reacting to seemingly chaotic business environment. To overcome this we need to increase the pulse of IT so that we’re operating at a higher pace than the business we support. Tools like LEAN software development have provided us with a partial solution, accelerating the pulse of writing software, but if we want to overcome this challenge then we need to find a new approach to managing IT.

Business, however, doesn’t have a single pulse. Pulse rate varies by industry. It also varies within a business. Back office compliance runs at a slow rate, changing over years as reporting and regulation requirements slowly evolve. Process improvement and operational excellence programs evolve business processes over months or quarters to drive cost out of the business. While customer or knowledge worker facing functionality changes rapidly, possibly even weekly, in response to consumer, marketing or workforce demands.

Aligning technology with business

We can manage each of these pulses separately. Rather than using a single approach to managing technology and treating all business drivers as equals, we can segment the business and select management strategies to match the pulse rate and amplitude of each.

Sales, for example, is often victim of an over zealous CRM (customer relationship management) deployment. In an effort to improve sales performance we’ll decide to role out the latest-greatest CRM solution. The one with the Web 2.0 features and funky cross-sell, up-sell module.

Only of a fraction of the functionality in the new CRM solution is actually new though—the remainder being no different to the existing solution. The need to support 100% of the investment on the benefits provided by a small fraction of the solution’s features dilutes the business case. Soon we find ourselves on the same old roller-coaster ride, with delivery running late,  scope creeping up, the promised benefits becoming more intangible every minute, and we’re struggling to keep up.

There might be an easier way. Take the drugs industry for example. Sales are based on relationships and made via personal calls on doctors. Sales performance is driven by the number of sales calls a representative can manage in a week, and the ability to answer all of a doctor’s questions during a visit (and avoid the need for a follow-up visit to close the sale). It’s not uncommon for tasks unrelated to CRM—simple tasks such as returning to the office to process expenses or find an answer to a question—to consume a disproportionate amount of time. Time that would be better spent closing sales.

One company came up with an interesting approach. To support the sales reps in the field they provided them with the ability to query the team back in the office, answering a clients question without the need to return to head office and then try to get back in their calendar. The solution was to deploy a corporate version of Twitter, connecting the sales rep into the with the call center and all staff using the company portal via a simple text message.

By separating concerns in this way—by managing each appropriately—we can ensure that we are working at a faster pace than the business driver we supporting. By allocating our resources wisely we can set the amplitude of each pulse. Careful management of the cycles will enable us to bring business and technology into alignment.

Innovation [2009-06-15]

Another week and another collection of interesting ideas from around the internet.

As always, thoughts and/or comments are greatly appreciated.

This issue:

  • Powering Ideas [Review of Australia’s Innovation System]
    Australia’s innovation agenda for the 21st century. Well worth the read
  • Who Drives Software Innovation? The “Best-of-Breed vs. Giants” Debate [SmartData Collective]
    Is it the industry giants or the smaller, and more agile, best-of-breed players that drive innovation in our industry?
  • The Future of the Workplace [Monocle]
    In the first edition of Monocle’s Design Dialogues, an intimate series of discussions on key design issues, they throw the spotlight on the future of the workplace.
  • How Twitter will change the way we live [Time]
    Are tools like Twitter changing the dynamics of innovation? Traditionally we have used metrics to measure innovation which capture the inputs to the productization process; numbers like volume of patents generated or size of R&D spend. As Steven Johnson says toward the end of this article, “if you measure global innovation in terms of actual lifestyle-changing hit products and not just grad students, the U.S. has been lapping the field for the past 20 years”. What should we measure (as what’s measure is what gets done) if we want to innovate like Twitter?

Innovation [2009-06-01]

Another week and another collection of interesting ideas from around the internet.

As always, thoughts and/or comments are greatly appreciated.

This issue:

  • The Disruption Talk [A VC]
    Fred Wilson (a VC in New York) partly crowd-sourced a presentation on disruption, presented at Google, recorded and stashed on YouTube.
  • In-N-Out Burger’s six secrets for out-and-out success [Daily Finance]
    On the heels of mounting cynicism generated by Wall Street bailouts and the perception that corporate leaders are gaming the system to make a profit, at least one American company is proving that businesses can survive and even thrive while sticking to traditional values.
  • Farmers Didn’t Invent Tractors. They Were Busy Farming. [Ben Casnocha]
    There’s a cliche in innovation / entrepreneurship which says, “Scratch your own itch.” That is, solve problems that you know really well. This is not always so.
  • Obama’s Seven Lessons for Radical Innovators [Harvard Business]
    Barak Obama’s presidential bid succeeded was a research lab for using social media in a political campaign. It differs from yesterday’s political organizations as much as Google and Threadless differ from yesterday’s corporations: all are a tiny handful of truly new, 21st century institutions in the world today. Here are the seven rules his campaign lived by.

The Scoop: Oracle swallows Sun

Gavin Clarke (Editor @ The Register), Rob Janson (President @ Enterprise Java Australia) and myself are on Mark Jones’ The Scope this week.

For loyal Sun customers and industry watchers, it was almost unthinkable – Oracle buying Sun. Sun Microsystems is one of Silicon Valley’s iconic technology companies, and Oracle doesn’t do hardware. And Sun was proud to wear the underdog badge. But the proposed acquisition raises fresh questions about the long-term health of the industry’s dominant suppliers. What’s the future hold for Oracle & Sun customers?

  • Oracle license inspections – costs to rise? What about Oracle’s famed licensing complexity. Will this get any better?
  • Consolidation problems: Will customer service deteriorate and product innovation wane?
  • What of Java – what’s Oracle likely to do with this prized jewel?
  • Did Oracle buy more problems than opportunities? (Sun’s debt, poor revenues…)
  • Enterprise app consolidation leaves CIOs with fewer choices: how will they bargain with suppliers now?
  • Larry Ellison said he wouldn’t buy Sun, or a hardware company, back in 2003. What changed? Does this mean that Oracle is likely to divest itself of Sun’s hardware business once the acquisition is completed?
  • What the growth engines for Oracle now – hardware/servers appear to have little headroom for serious growth.

About The Scoop

The Scoop is an open, free-flowing conversation between industry peers. It’s about unpacking issues that affect CIOs, senior IT executives and the Australian technology industry. The conversation is moderated by Mark Jones, The Scoop’s host and producer. More information about The Scoop, including a list of previous guests, can be found here:

http://filteredmedia.com.au/about-the-scoop/

What we’re doing today is not what we did yesterday

Telxon
Telxon hand unit

The business of IT has changed radically in the last few years. Take Walmart for example. In the 80s Walmart laid the foundations for its future growth by fielding a supply chain data warehouse. The insight the data warehouse fueled their amazing growth to become the largest retailer in the world. However, our focus has moved on from developing applications. More recently Walmart fielded the Telxon, a barcode scanner with a wireless link to the corporate back-end. This device is the front end of a distributed solution which has let Walmart devolve buying decisions to the team walking the shop floor.

For a long time IT departments have defined themselves by their ability to deliver major applications into the enterprise. CRM, MRP, even ERP; all the three letter acronyms. For a long time this has been the right thing to do. Walmart’s data warehouse, to return to our example, was a large application which was a significant driver in the company’s outlier performance for the next couple of decades.

The world has changed a lot since that data warehouse went operational. First the market for enterprise applications grew into the mature market we see today. If you have a well defined problem—an unsupported business activity—then a range of vendors will line up to provide you with off-the-shelf solutions. Next we saw a range of non-technology options emerge, from business process outsourcing (BPO) and leveraging partnerships, through to emerging software-as-a-service (SaaS) solutions.

What used to be a big problem—fielding a large bespoke (or even off-the-shelf) application—has become a (relatively) small one. Take CRM (customer relationship management) as one example. What was a multi-year project requiring an investment of tens of millions of dollars to deploy a best-of-breed on-premises solution, has become a few million dollar and a matter of months to field SaaS solution. And the SaaS solutions seem to be pulling ahead in the feature-function war; Salesforce.com (one of the early SaaS CRM solutions) is now seen as the market leader (check with your favorite analyst).

Nor has business been standing still while technology has been marching forward. The productivity improvements provided by the last generation of enterprise applications have created the time and space for business stakeholders to solve more difficult problems. That supply chain solution Walmart deployed that was the first of many, automating most (if not all) of the mundane tasks across the supply chain. Business process methodologies such as LEAN (derived from the Toyota Production System) and Six Sigma (from GE) then rolled through the business, ripping all the fat from our supply chains as they went past. The latest focus has been category management: managing groups of product as separate businesses and, in many chases, handing responsibility for managing the category back to the supplier.

Which brings us back to the Telxon. If we’ve all been on the same journey—fielding a complete set of applications, optimizing our business processes, and deploying the latest, best practice, management techniques—then how do we differentiate? Walmart realized that, all things being equal, it was their ability to respond to supply chain exceptions that would provide them with an edge. As a retailer, this means responding to stock-outs on the shop floor. The only way to do this in a timely manner is to empower the people walking the floor to make a procurement decision when they see fit. Walmart’s solution was the Telxon.

The Telxon is an interesting device as it reveals an astonishing amounts of information: the quantity that should be on the shelf, the availability from the nearest warehouse, the retail price, and even the markup. It also empowers the employee to place an order for anything from a pallet to a truck-load.

Writer
Writer Charles Platt during his stint as a Wal-Mart employee in Flagstaff, Ariz.

As one journalist found:

We received an inspirational talk on this subject, from an employee who reacted after the store test-marketed tents that could protect cars for people who didn’t have enough garage space. They sold out quickly, and several customers came in asking for more. Clearly this was a singular, exceptional case of word-of-mouth, so he ordered literally a truckload of tent-garages, “Which I shouldn’t have done really without asking someone,” he said with a shrug, “because I hadn’t been working at the store for long.” But the item was a huge success. His VPI was the biggest in store history—and that kind of thing doesn’t go unnoticed in Arkansas.

Charles Platt, Fly on the Wall (7th Feb 2009), New Your Post

Clearly the IT world has moved on since that first data warehouse went live in Arkansas. Enterprise applications have been transformed from generators of competitive advantage into efficient sources of commodity functionality. Technology’s ability to create value should be focused on how we effectively support knowledge workers and the differentiation they create. These solutions only have a passing resemblance to the application monoliths of the past. They’re distributed, rather than centralized, pulling information from a range of sources, including partner and public sources. They’re increasingly real time, in the Twitter sense of the term, pulling current transactional data in as needed rather than working from historical data and relying on overnight ETLs. They’re heterogeneous, integrating a range of technologies as well as changes in business processes and employee workplace agreements, all brought together for delivery of the final solution. And, most importantly, they’re not standalone n-tier applications like we built in the past.

But while the IT world has moved on, it seems that many of our IT departments haven’t. Our heritage as application factories has us focused on managing applications, rather than technology, actively preventing us from creating this new generation of solutions. This behavior is ingrained in our organizations, with a large number of architects through project managers to senior management measuring their worth by the size of the project (in terms of CAPEX and OPEX required, or head count) that they are involved in, with the counter productive behavior that this creates.

In a world where solutions are shrinking and becoming more heterogeneous (even to the extent of becoming increasingly cross discipline) our inability to change ourselves is the biggest thing holding us back

Innovation [2009-05-18]

Another week and another collection of interesting ideas from around the internet.

As always, thoughts and/or comments are greatly appreciated.

This issue:

We’re making our lives too complicated

Has SOA (Service Oriented Architecture) finally jumped the shark? After years of hype and failed promises, SOA seems to be in trouble. In a few short months it’s gone from IT’s great savour to something some people think is better forgotten.

The great promise of SOA was to deliver an IT estate which is more agile and cost effective than was possible with other, more conventional, approaches. By breaking our large problems into a set of much smaller ones, we would see more agility and a low total cost of ownership. The agility would come from the more flexible architecture provided by SOAs many moving parts. A lower cost of ownership would come from reuse of many of these moving parts. Many companies bought into this promise, and started major SOA transformation programs to “SOA enable their business”. Once the program of work was delivered they would have a shiny new flexible, and cost effective IT estate. The business would be thrilled, and the old tensions between business and IT would just melt away. More often than not the business wasn’t thrilled as the program failed to deliver the promised benefits.

The problem, it seems, is that we’re focused on creating cathedrals of technology. Cathedrals were the result of large bespoke development efforts. The plans often consisted of only a rough sketch on a scrap of paper, before a large number of skilled craftsmen were engaged. The craftsmen broke the problem into many small parts that were then laboriously assembled into the final structure, often adjusting the parts to fit in the process. While this process created a number of spectacular buildings, the journey from initial conception to completed build was long and challenging.

The lack of engineering pragmatism frequently resulted in cathedrals collapsing before they were finished, often multiple times. The reason we know that a flying buttress worked was because it hadn’t failed, yet. People died when a structure collapsed, and there was no way of telling if the latest version of the structure was about to collapse. The lengthy development process often lasted generations, passing through the stewardship of multiple architects with no clear end in sight. Many cathedrals, such as the one in New York, are still considered unfinished.

A lot of SOA projects give off a strong smell of cathedral. They are being constantly re-architected—while still in development—to cope with the latest business exception or demand. When they’re introduced to the hard reality of supporting solutions bits of them collapse and need to be rebuilt to support our new (and improved) understand of what will be demanded of them. And, finally, many of them are declared “finished” even though they are never fully baked, just so we can close that chapter in our company’s history and move onto the next problem.

Modern approaches to building construction take a different approach. A high level plan is created to capture the overall design for the building. The design is then broken into a small number of components, with the intention for using bespoke craftsmen for the fine details that will make the building different, while leveraging large, commoditized, pre-fabricated components for the supporting structures that form the majority of the building. Construction follows a clear timetable, with each component—from the largest pre-fabricated panel through to the smallest detail—integrated into the end-to-end solution as it is delivered. Complexity and detail were added only where needed, with cost effective commoditized approaches minimizing complexity elsewhere. A clear focus on the end goal is maintained throughout the effort, while clear work practices focused on delivering to the deadline ensure that the process was carried out with a minimum of fuss (and no loss of life).

The problem, it seems, is that we’re confusing agility with flexibility. The business is asking for agility; the world is changing faster than ever and the business needs to be in a position to react to these changes. Agility, or so our thinking goes, requires flexibility, so to provide a lot of agility we need to provide a lot of flexibility. Very soon we find ourselves breaking the IT estate (via our favorite domain model) into a large number of small services. These small parts will provide a huge amount of flexibility, therefore problem solved!

This misses the point. Atomizing the business in this way creates overhead, and the overhead soon swamps any benefit. The effort to define all these services is huge. Add a governance process—since we need governance to manage all the complexity we created—and we just amplify the effect of this overhead. Our technically pure approach to flexibly is creating too much complexity, and our approach to managing this complexity is just making the problem worst.

We need to think more like the architect of the modern prefabricated building. Have a clear understanding of how the business will use our building. Leverage prefabricated components (applications or SaaS) where appropriate; applications are still the most efficient means of delivering large, undifferentiated slabs of functionality. And add complexity only in those differentiating areas where it is justified, providing flexibility only where the business needs. In the end, creating good software is about keeping it simple.  If it’s simple, it gets done quickly and can be maintained more readily.

Above all, favor architectural pragmatism over architectural purity. The point of the architecture is to support the business, not to be an object of beauty.

The Value of Enterprise Architecture

Note: Updated with the slides and script from 2011’s lecture.

Is Enterprise Architecture in danger of becoming irrelevant? And if so, what can we do about it?

Presented as part of RMIT’s Master of Technology (Enterprise Architecture) course.

The Value of Enterprise Architecture