Be careful what you measure

Tyler Cowen has an article over at MIT Technology Review, Measured and Unequal, that discusses how improved measurement of workers might be a fundamental driver of inequality in the workplace of the future.

Consider journalism. In the “good old days,” no one knew how many people were reading an article like this one, or an individual columnist. Today a digital media company knows exactly how many people are reading which articles for how long, and also whether they click through to other links. The exactness and the transparency offered by information technology allow us to measure value fairly precisely.

The result is that many journalists turn out to be not so valuable at all. Their wages fall or they lose their jobs, while the superstar journalists attract more Web traffic and become their own global brands. Some even start their own media companies, as did Nate Silver at FiveThirtyEight and Ezra Klein at Vox. In this case better measurement boosts income inequality more or less permanently.

The assumption behind this sort of piecework measurement is that all the value realised by an article is due to the sweat and toil of a more-talented-than-usual journalist. If your article gets the clicks, then it must be because you are so good at what you do.

Unfortunately the world is not so simple.

We might choose to build our organisations around this sort of idea (and indeed, BuzzFeed et al work this way) but it tends to foster a short term and overly transactional view of work that ignores a lot of the value that workers, or a community of workers might create.

The first problem is the obsessive focus on outputs, on the assumption that the worker is responsible for all the value created. Outputs depend on inputs, and not just the worker’s skills. You can’t make a silk purse out of a sow’s ear, as the saying goes.

While the worker might be skilled, their work is also dependant on the quality of the materials they have to work with. Take the journalism example. A manager somewhere is splitting up the work, either by handing out the story ideas or by allocating topics to individuals. Not all ideas or topics are equal. It’s possible for someone to come from outside this system by finding a new approach—as Nate Silver did with a data-drivern approach—but that’s the exception rather than the rule. It’s more typical for the quality of the value of the outputs to be bound by the quality of the inputs, not the effort of the individual.

We see something similar in sales. It’s easy to sell in a rising market, and a booming market will see many sales people getting large commissions for no reason other than turning up. In a down market, though, its a different story, and we punish some of our best people for working hard just to bring anything into the business.

If we want to reward individuals based on their contribution then we need to quantify the amount of value they added, rather than the amount of value they lucked into. If we don’t then we’ll create a feeding frenzy for the juicy bits of work, while other less attractive (but possibly no less important in the overall scheme of things) get ignored.

Unfortunately it’s surprisingly difficult to measure value-add for many workers as it can be challenging to gauge the quality of the materials that they have to work with. A good example of this are the efforts in the US to measure teachers on the value they add in the class room, efforts which are struggling as it seems nearly impossible to objectively measure the quality of the students that they have to work with. There’s just too many variables.

Second is the problem of cumulative advantage. Success typically brings more success for no other reason than you were successful. Consider the opportunities created when you win an Oscar. The Oscars are an annual competition, so they’re awarded even if the year’s releases aren’t particularly good (such as if there’s a writers strike during most of the past year).

It doesn’t matter how you win the Oscar—either by creating great art and a big box office success, or simply be being the best of a bad lot—the attention that the Oscars garners you brings you to the attention of the world and the opportunities start flowing in. This improves the quality of the materials you can choose to work with. You might break the VW emissions story due to dumb luck, but it results in more story ideas flowing your way. You might not be the best journalist, you might not even be the journalist best positioned to make the most of the idea, but the idea is yours none the less.

Entire careers are built on the back of a lucky break followed by cumulative advantage. While this is good for the few lucky individuals, it’s not so good for the firm as it means that the firm might not be making the most of the materials at its command (though picking winners does make it easier for management). Nor is it much good for the equally talented individuals who weren’t quite so lucky.

Third is the problem of context. It’s rare, these days, to work in isolation. The context we’re in provides us with resources and connections that we couldn’t get elsewhere, or even just a boss that we can work with. While we might thrive in one environment, we struggle in others. One good example is star analysts, who often struggle when they leave the firm where they built their reputation. Some of that value in the outputs created might be the result of a productive work culture or effective management structure and team, factors that are the result of the everyone’s contributions, and not just the contributions the individual creating the deliverable.

Mr Cowen’s problem is that he has mistaken ease for cost. It’s cheaper than ever to measure all sorts of factors associated with work. At the same time, work has evolved making it hard know what to measure. While it might be cheap to generate all sorts of stats on worker activity, it’s not easy to tie these back to productivity.[ref]Aside, that is, for work situations which are explicitly configured as piece work, such as Uber drivers.[/ref]

The root cause of this a recent shift (possibly sometime around 2005) from value being defined by the producer, to being defined by the consumer. The emergence of the consumer internet put the consumer in control as it enabled the consumer to have more information on a product than the merchant or producer, and the ability to source the product from any merchant around the globe. This was followed by the more recent emergence of social media, enabling consumers to turn to their peer, rather then brands.

Value used to be defined in terms of product features and functions, and we could measure a worker’s productivity by their contribution to creating these features and functions. Frederick Taylor started the trend by measuring how long it took for a man to unload a cart. The modern version is the basis of Mr Cowen’s article: counting the number and reach of articles carrying a byline, or worker surveillance where everything a worker types at a computer, everything they do is logged, recorded, and measured.

Value today is defined by a customer’s relationship to a product. Value is relative and shifting because it is a function of an expanding choice space for consumers. While all your workers contribute to creating this value, it’s not always obvious how to quantify their contribution.Their contribution might also be different for each customer, as relative value means that each customer could possibly conceive value differently.

Any retailer who heads down the omnichannel path, for example, needs to deal with the challenging of aligning a salesforce measured on their sales with a strategy that has sales skipping across multiple channels and contact points as the customer learns about the firm, develops their own understanding of what value is created, and winds their way to a decision. When you consider this it’s not surprising the Apple’s stores (some of the most profitable in the world) are not measured on sales, and fall under the marketing budget.

In the mean time we have many firms racing to quantify and optimise individual tasks that their workers undertake. This might drive improvements in a short term and overly instrumentalist definition of productivity, and result in a few lucky individuals receiving large pay checks. In the longer term the same strategy is destroying the value created for the customer, and possibly taking the firm’s future with it.

Image: Lainey Powell