The Perils of Metrics

Got blood?  Here’s an elegant example of the positive power of data-informed decision-making. Transfusions save lives, but they are also expensive and not without risks.  Each decision whether or not to transfuse is the output of some decision-making process, presenting the opportunity to reach a better balance by improving the algorithm.  It may one day… Continue reading Dr. Bayes will see you now

Let’s start the year with an example that is, at least superficially, on the light-hearted side:  librarians creating fake patrons to check out important books that would otherwise be discarded for lack of popularity.Underneath are two deep and important dysfunctions.  The specific one — aiming to show people only what they are likely to want… Continue reading Does Nicolas Bourbaki read Charles Dikkens?

Here’s an interesting way to take metrics dysfunction to an entirely different level.  Examples abound of optimizing for only easy-to-measure aspects of an activity and getting, well, what was measured.  The new and innovative idea described here is to select among activities based on which of them have easy-to-obtain metrics.  Another way to look at… Continue reading Game the metrics? Fluxx that!

During a presidential debate in 1980 Ronald Reagan famously asked “Are you better off than you were four years ago?”.  His next sentence is less well-known but it shows how quickly we move to conflate measurement of human well-being with financial and economic metrics:  “Is it easier for you to go and buy things in… Continue reading Even a googol zeros add up to nothing

If Wells Fargo had set out to write a cautionary tale, a case study of data-driven dysfunction, they could not have succeeded better. The first article posted here showed how creating strong incentives to hit numerical targets drove people over the edge.  Entirely predictable, and straight out of Austin’s Measuring and Managing Performance in Organizations. … Continue reading Lies, damn lies, and data science

Wells Fargo paid $185M in fines and 5,300 people paid with their jobs to bring you this perfect example of metrics dysfunction.   The question that does not seem to have been answered is whether or not the people who set up the system — the people who put 5,300 line workers in the impossible… Continue reading Stagecoach robbery

Careful, thoughtful, painstaking use of metrics; clear aspirational goals; intrinsically motivated people with no desire to game the system:  all together, a big improvement.   “For five years Charlie took it upon himself to create a new workflow system for the tracing center, breaking down each step in the tracing process into equations, doing time-motion… Continue reading Ready, aim, fire! A metrics bulls-eye!

A large and possibly calamitous financial hole for the aged and infirm at the intersection of two metrics tied tightly to incentives.    Flexibility and discretion in the application of rules invites abuse and corruption.  Inflexible application produces unintended consequences and invites efforts to game the rules.   http://www.nytimes.com/2016/08/07/us/politics/new-medicare-law-to-notify-patients-of-loophole-in-nursing-home-coverage.html?smid=go-share

“The main measure of economic activity, GDP, counts housework when it is paid, but excludes it when it is done free of charge. This is an arbitrary distinction, and leads to perverse outcomes. … The usual defence is that measuring unpaid work is hard.”   The perverse outcomes are in many ways visited upon women,… Continue reading It’s not work if you don’t get paid for it

“`The p-value was never intended to be a substitute for scientific reasoning,’ the ASA’s executive director, Ron Wasserstein, said in a press release.”   Note that p-values are metrics, measures of a particular probability under certain assumptions.  As usual, the Goal – Question/Signal – Metric framework is appropriate:  the goal would be to determine, say,… Continue reading What’s the value of this value?