Predicting productivity
To follow up on the discussion in my last post about testing regimes, I want to recommend a New Yorker article by Malcolm Gladwell about evaluating hard-to-evaluate skills. He describes a football scout named Dan Shonka working in his hotel room carefully reviewing the DVD of a game he has just watched live: “He had a stack of evaluation forms next to him and, as he watched the game, he was charting and grading every throw that Daniel [the quarterback] made…. Shonka had seen all the promising college quarterbacks, charted and graded their throws, and to his mind Daniel was special: ‘He might be one of the best college quarterbacks in the country.’ ” The football industry in the US also collects mountain of statistics on each player. Imagine just going with the objective statistics and losing the subjective evaluation of the scouts. Gladwell says that even with people like Shonka watching rising quarterbacks like a hawk, the failure rate is high. Some productivities are just very hard to assess and predict before the fact.
“We now realize that being a good doctor requires the ability to communicate, listen, and empathize—and so there is increasing pressure on medical schools to pay attention to interpersonal skills as well as to test scores.” Every line of work requires skills that are difficult to test, or determine ex ante. Every army in the world tries to recruit the best fighters, but only in the battlefield true heroes can be told apart from mere soldiers. James Heckman, the 1998 winner of the Nobel Prize in Economics, is noting a similar problem when he speaks of the “staggering gap between the list of productivity characteristics available to economic analysts in standard data sources and what is available to personnel departments of firms.”
Economists David Berri and Rob Simmons analyzed the performance of quarterback in the professional league based on test scores and how they played their college games. They found no relation. The effectiveness of football scouts was limited not because they did not have the skill to measure college quarterback quality, but that “the kind of accuracy required to do the job well could be measured only in a real N.F.L. game. (NFL stands for the National Football League, which is the premier professional league in the US.)
What does all this mean for testing regimes and productivity? The upshot of the points raised by Gladwell is that like heroism that is only tested on the battlefield, productivity is only observable on the job, and by the employer. This does not mean that the US system of “employment at will” is the best system, it only means that when someone other than the employer tries to judge productivity, a testing machine or a regulator, something is lost.
Tyranny of numbers 
By the way, I think the main point of the article was this: in situations where there is no good indicator for selection by information available before it, let as many as possible to try it in real situations and have the gate as open as possible, then based on their performance throw those performing weak out of system.
On more thing, I think the situation about student selection can be considered even more complicated than selecting teachers, since the real testing situation will come after the university, in the job market (if we assume that the main point of universities is teaching people suitable for feeling jobs in the market, though I think this is not the only objective). The article does not deal with this kind of situation.
I also think that there is a major difference between undergraduate and graduate programs. In the later, it is possible to have a better testing environment, either though research results or through the internships and real projects carried out, though it seems that universities and advisers in Iranian universities do not have enough time and skill (with the exception of top universities) for this work.
An I still think that the direct link between between the degree and income, without taking the quality of work done, is a major obstacle. A person with a BS may carried out a job better than a person with a MS, but unfortunately, the later is required by law to be payed more.
I have finally found some free time to read the article you linked. I think if we want to apply the main point of the article to student selection proses naively, we have to let almost all of 1.5 million prospective students to enter Sharif’s EE! A more reasonable approach would be to change the system such that getting in is easy but getting out is hard. I feel that almost all of students entering university in Iran’s top universities get their degree, the hard part is getting in the university, whereas where I’m right now (one of top North American universities), a large number of undergrads drop each year. I will look for numbers but from what I have heard, less than %25 finish in our department, and I guess the situation is similar in other departments. More than %20 drop each year. Compare this to Sharif, where more than %90 get their degrees. Maybe it has to do with Iranians feeling that they have the right to get their degree by entering a university, whereas this is not correct in North America. What you think?
I am glad you are interested in this topic and thanks for reading my paper. If the paper you have in mind with Murphy, you misunderstand it. You can have the same admission rate but evaluate students on a wider set of skills to increase productivity. The point of that paper and my arguments here is that testing with multiple choice limits skills acquisition for the average students in ages 0-18!
Also, about graduation rates in the US do check your numbers, you will be surprised!
Sorry, I have not read it yet, I was talking about the NYTimes article. I am going to read your article after this election thing ends. 🙂 By the way, a few interesting links that may find useful:
http://www.primaryreview.org.uk/
http://www.cre.org.uk/
http://www.dcsf.gov.uk/primarycurriculumreview/
http://publications.teachernet.gov.uk/default.aspx?PageFunction=productdetails&PageMode=publications&ProductId=DCSF-00499-2009&
[…] Excerpt from: Predicting productivity « Tyranny of nu… […]