In his book, Thinking Fast and Slow, Nobel prize-winning behavioural economist Daniel Kahneman points out, “Several studies have shown that human decision makers are inferior to a prediction formula even when they are given the score suggested by the formula!” He uses as an example an algorithm that predicts the future price of Bordeaux wines using three weather-related variables, which has an accuracy of .90!
It might seem surprising that a formula can make better predictions than a human expert. After all, we know that computers are not as “smart” as people are. They can’t learn the way we do, make associations, accommodate to changing rules, or get the concept of humour. They do, however, make mathematical calculations much faster than people can, more accurately, and over much larger sets. And they don’t second-guess themselves over the results.
So let’s entertain the idea that a formula can make accurate predictions, given a set of consistent inputs. How might we use this to our advantage? What about using an algorithm to analyze a large set of records and divide it into more manageable chunks? Then, a person can use his or her knowledge to further analyze the smaller set and decide what to do with them. Here’s my experience in a real world scenario.
One of my clients is the director of an art department that designs product packaging. In this particular department, we use FileMaker to manage the overall job workflow using tasks which are clicked off by users when done. Based on when the process is begun, it’s possible to project end dates for each task and for the project as a whole. It’s also possible to compute the length of time a task is overdue, and to relate tasks to one another in terms of elapsed time.
The traffic manager is in charge of routing jobs to the production artists in order of priority. She sets the priorities and places the job folders in a “hopper” placed in a central location with the most urgent at the front. The next available designer picks up the first one in the queue.
My client wanted to develop a simple system that would let the traffic manager predict which projects are likely to become urgent. For instance, if a job comes in without all the required information, or if there is a delay at a critical point in the process, it is easy to see that the job is likely to completed more slowly than a job that arrives with complete information and has no delays. This is not so difficult to determine when dealing with a handful of jobs, but with tens or hundreds of them it becomes much harder to keep all the priorities in your head.
What he came up with was to use an algorithm. Based on criteria already being gathered through the task dates and job information, we decided to look at a number of variables and assign a simple number score to each one. Then we’d add up all the variables to get a total for the job. The higher the score, the more urgent the job.
At first, it seemed like a difficult task to sort out what variables should be tracked. There was initially much resistance from the traffic manager, who had a sense that it was too complicated, and couldn’t be done by computer. However, people use rules too — we just don’t think of them that way — in decision-making, and the fact is that a computer can track many more variables, more accurately than a person can (see above).
We started by asking the traffic manager how she was currently assigning the priorities. A few items came to mind right away and we wrote those down. As we ran out of ideas, we went over to the actual hopper and looked at the jobs and started examining the reasons why certain jobs had been placed before other ones. This generated a lively discussion and a few more rules. It started to become obvious that there was indeed a method to the madness. Though it was somewhat complex, it was possible to nail down what had started out as rather vague criteria.
The Technical Stuff
In the end, there are seven variables being assigned a score (though you can probably get good results with three variables). The scoring calculation is set so that jobs get no score at all unless certain conditions are met. Some variables get a higher score than others, which has the effect of weighting some more heavily. For instance, most of the variable scores are between 1 and 5, but if the job is marked explicitly as urgent, then a 10 is added. This ensures that urgent projects, whatever the reason, are always at the top of the list when it’s sorted by score in descending order.
Here’s a simplified version of what’s being evaluated:
• Final Art Assigned date > 1 day (1)
• Disk to Printer date is past (2)
• Job type (3)
• Hard Launch Y/N (4)
• Number of pieces of artwork in the job (2-4)
• Senior Management artwork approval Y/N (5)
• Urgent Y/N (10)
I won’t go into the reasons for the various scores and the specific calculations, which isn’t important to understanding the concept. The values we assigned to the variables are arbitrary, other than their relative value to one another, and we can always adjust them up or down if necessary. Adding up all the individual scores, you can see that the highest score a job can have is 19.
Looking at the actual records, the median score is about 8 for work in progress jobs. But depending on the mix of jobs and their state of completion, there are a few jobs that score 12-15. These constitute the red-flag, urgent projects that need attention, and they’re pretty easy to see at the top of the list.
What Else is it Good For?
Here’s another benefit of looking at the scoring in aggregate. The more jobs there are at the top with high scores, the more urgent work there is in the department. This could indicate a serious backlog!
It also turns out to be useful to look at the score breakdown as well, since the total doesn’t give you too many clues about what exactly might be going wrong. If the traffic manager sees the score for a job creeping up, she has the tools to investigate a little further and see what’s happening with it.
This is an example of a computer algorithm doing a form of triage on a large set in order to reduce the number of records to a more human size. The algorithm is ordering possibly hundreds or thousands of records by many more variables than a person can hold in their mind, more consistently and reliably. It’s similar to how Google determines what results might be most relevant to your search term, or how Facebook suggests who else you might know based on the number of mutual friends you have.
The human user still has the option to do something different if they want, for instance by marking something urgent or assigning a job directly without regard for the score. But in the vast majority of cases attention is focused on the top handful of jobs that need action. As it turns out, the traffic manager kept an open mind and is now using the system quite happily.
I can see lots of applications for this idea. The calculations could be adapted to indicate importance or relative size instead of urgency, for instance. It’s important to choose the variables carefully. It could be a challenge to figure out what to track, how many variables to track, and what scoring regime seems reasonable.
Applying an algorithm lets the computer do what it’s good at, which is crunch numbers. After all, it’s not about endless columns of data, but about information YOU can use. Can you think of other ways to use this idea?