To follow up on the discussion in my last post about testing regimes, I want to recommend a New Yorker article by Malcolm Gladwell about evaluating hard-to-evaluate skills. He describes a football scout named Dan Shonka working in his hotel room carefully reviewing the DVD of a game he has just watched live: “He had a stack of evaluation forms next to him and, as he watched the game, he was charting and grading every throw that Daniel [the quarterback] made…. Shonka had seen all the promising college quarterbacks, charted and graded their throws, and to his mind Daniel was special: ‘He might be one of the best college quarterbacks in the country.’ ” The football industry in the US also collects mountain of statistics on each player. Imagine just going with the objective statistics and losing the subjective evaluation of the scouts. Gladwell says that even with people like Shonka watching rising quarterbacks like a hawk, the failure rate is high. Some productivities are just very hard to assess and predict before the fact.
“We now realize that being a good doctor requires the ability to communicate, listen, and empathize—and so there is increasing pressure on medical schools to pay attention to interpersonal skills as well as to test scores.” Every line of work requires skills that are difficult to test, or determine ex ante. Every army in the world tries to recruit the best fighters, but only in the battlefield true heroes can be told apart from mere soldiers. James Heckman, the 1998 winner of the Nobel Prize in Economics, is noting a similar problem when he speaks of the “staggering gap between the list of productivity characteristics available to economic analysts in standard data sources and what is available to personnel departments of firms.”
Economists David Berri and Rob Simmons analyzed the performance of quarterback in the professional league based on test scores and how they played their college games. They found no relation. The effectiveness of football scouts was limited not because they did not have the skill to measure college quarterback quality, but that “the kind of accuracy required to do the job well could be measured only in a real N.F.L. game. (NFL stands for the National Football League, which is the premier professional league in the US.)
What does all this mean for testing regimes and productivity? The upshot of the points raised by Gladwell is that like heroism that is only tested on the battlefield, productivity is only observable on the job, and by the employer. This does not mean that the US system of “employment at will” is the best system, it only means that when someone other than the employer tries to judge productivity, a testing machine or a regulator, something is lost.