When we work with organizations to build scorecards, we always stress that those that work with measures need to understand at least the basics of how measures work. In assignments, we often have to explain the importance of understanding confidence levels and intervals when using surveys. This is not rocket science.
With at least the basics understood, we then progress to the basics of ana-lytics. In time, they can mature to a more advanced understanding of mea-surement and analytics. But simply understanding the basics means that organizations do not spend endless amount of times collecting KPIs and then providing commentary that is at best of limited value or (not uncommon) downright dangerous as the so-called analysis leads to strategic, and often expensive, improvement interventions that are not addressing the problem—
and perhaps exacerbating it.
It is a continued mystery how organizations are, as is typically the case, obsessed with measurement, but do not invest the time and money into teach-ing those that work with measures even the basics of the underpinnteach-ing sci-ence. We would expect a finance professional to understand finance and the same for an IT specialist, but not for those working with KPIs. We need to redress this odd, and dangerous, omission.
The Dangers of Aggregation
Of the many common mistakes that organizations routinely make, consider aggregation.
Here’s a task: put one leg in a bucket of boiling water and the other in a bucket of freezing water. On average, it’s the perfect temperature. Herein lies a major issue with measurement that we commonly observe: believing that aggregated data is an insightful measure of performance. Of course, aggrega-tion has some value as a high-level performance indicator, but without inter-rogating, the data beneath it can be very misleading.
Simpson’s Paradox
As a powerful illustration, consider Simpson’s Paradox: the paradox in prob-ability and statistics, in which a trend appears in different groups of data but disappears or reverses when we combine these groups.
As a true example, a University in the USA was taken to court by a young woman that claimed gender bias on the basis that the annual admission data showed that significantly more boys were being admitted than girls. Sounds fair, yes?
However, the analysis of the data showed that generally girls were applying for the most competitive courses, whereas boys were more attracted to the less competitive courses. In reality, more girls were being admitted to both the more competitive and less competitive courses. Yet, when the numbers were aggregated, there were more boys admitted. Simpson’s Paradox.
Consequently, whenever we are shown aggregated data (which on most scorecards are colour-coded), we always ask “but what does this mean?”
Typically, the answer we get is “It shows we are performing well”—if it’s green.
Maybe it does or maybe it doesn’t. We have no idea without looking at the underlying data. The downsides of colour coding are explored in Chap. 9:
Unleashing the Power of Analytics for Strategic Learning and Adapting.
Herein lies another major issue with measurement: the belief that the reported KPI score is sufficient information for decision-making purposes. It is not. The top-level KPI “number” does not provide the full picture of perfor-mance. Back to the word “indicator.”
Trend Analysis
A further area of concern is a failure to understand trends properly. It is the trend that is important, not the direct comparison between two adjacent num-bers. This enables, as one example, performance to be analysed so to under-stand if any variations to performance can be attributed to reasons such as normal seasonal change or is due to negative influences that to redress require targeted interventions. As one practitioner noted, “We can make informed choices based on analysis that tell us that yes performance to a target has fallen by 10% but it’s not significant so there’s no need to do anything about it.”
Moreover, good trend analysis might show that although a financial KPI is still green, it is trending downward, whereas a red KPI is trending upward and is forecasted to move into yellow soon. Much better to intervene to stop the downward trends than the one trending upwards. This points to one of the shortcoming of exception-based reporting, where all the conversations are about “apparently” underperforming KPIs.
Driving “Rational” Behaviours
A poor understanding of the science of measurement also means that organi-zations overlook the fact that measurement does not always drive the expected behaviours.
An Amusing Story
Here’s an amusing true story. A few years back, the city of Mumbai had an issue with rat infestation. The rat population was growing rapidly and, of course, causing all manner of health concerns. Then a government official had a great idea. Mumbai also has a lot of poor people, so simply pay people to kill rats—deliver X numbers of dead rats and receive X rupees…brilliant!
It worked beautifully. The rat population declined rapidly. Nevertheless, after a few months it started to rise again, and the number of dead rats deliv-ered for payment rose as well. No-one could understand why. Investigations
found the answer—people were breeding rats. Makes sense as rats breed quickly, and they were proving a useful source of income to struggling folk.
A Tragic Story
The next true story is not amusing. A pizza company had an objective “Deliver Superior Customer Experience” and a supporting measure of “95% of Pizzas delivered within 15 minutes.” This was based on research that found that customers wanted their food hot and delivered quickly. Makes sense, and to encourage the delivery of this customer experience, outlet managers’ bonuses were based largely on this KPI.
One day, an outlet had an issue with its ovens and there was a panic about not hitting the target. The manager told a delivery boy to get on his motor-cycle, hurry up, and get to the customers’ homes quicker than he normally would have so as to hit the target. The young driver drove quickly, crashed, and died.
Be Careful What You Ask For
Both these amusing and tragic stories deliver the same message. Be careful what you ask for when creating KPIs and setting targets. They might just encourage “rational” behaviours that could be either positive or negative.
Dysfunctional behaviours (which are simply “rational” responses—that is, doing what is required to hit the target) triggered by a KPI are far from uncommon. It is well known (and we have seen this) for a manufacturing plant to bet set a target to reduce reported injuries, and for the target to be reached simply by only reporting serious injuries (that cannot be hidden).
Performance does not improve, but the target is met. Put another way, “the target is hit, but the point is missed.” There are many similar examples from all sectors, industries, and functions.
Identifying Rational Behaviours
When we work with organizations to select KPIs and targets, a step we always include is to get people to think about the behaviours the KPI might drive.
We simply ask them to brainstorm and write down all the positive behaviours that might be encouraged and then the negatives. When done, we then dis-cuss how to best encourage the former and mitigate the latter. Sometimes, the
risk of dysfunctional behaviour is so great that the KPI must be rethought or abandoned: a simple exercise that can deliver a lot of benefits and save a lot of heartache.
Sometimes negative behaviours can happen as the KPI transitions from the design phase to reporting. We worked with one government organization that had set a KPI for the finance department of “90% of invoices paid within two months.” This was something of a stretch for a government entity in that country. So, when we reviewed the finance scorecard, we were surprised to find the colour was green. It was being hit. This made little sense as in conver-sations with suppliers, a common gripe was that it took up to eight months to be paid. Clearly, something was amiss.
Auditing the KPI found that although the original intent was payment within two months of receipt, finance had changed this to two months from final sign-off, which—in this very bureaucratic organization—took six months. Again, performance did not change but the target was hit. No need for exception reporting here.
So, when designing KPIs and targets, think about the rational behaviours (positive and negative) that might be encouraged and plan accordingly. Also, ensure that the original performance-enhancing intent of the measure is not changed (oftentimes surreptitiously) during implementation. Indeed, a regu-lar audit of the Balanced Scorecard is good practice (and when managers resist this, it is a strong indication that something just ain’t right).
In addition, be particularly careful when bonuses are linked to KPI target achievement. An old adage says, “What gets measured gets done. What gets rewarded gets repeated.” Be careful you don’t simply end up rewarding more rats.
Advice Snippet
Organizations make a number of mistakes when working with KPIs. Amongst the most common are:
• A failure to understand the potential dangers of aggregating data. As well as hiding potentially damaging performance trends (hidden in the measures that are aggregated), they also might give a totally misleading view of per-formance: Simpson’s Paradox.
• Not taking confidence levels and intervals into account leads to wasting time discussing statistically meaningless data. Best practice is to be 95% confident that the figure provide is correct to an error rate of two percentage points.
• Simply comparing one data point with the one previous. This provides a performance snapshot and is only meaningful when the organization has