Peer review

Definitions Peer review is the name of a process by which the work and ideas of an individual or group is assessed by another individual or group considered to have a level of expertise near to that of the assessed. Thus the reviewers are deemed to be the "peers" of the assessed. The purposes of peer review are to inform decisions either on the allocation of funds among a number of applicants ("research grant agency peer review "), or on the publication of the results of research ("editorial peer review"). This web-page is concerned primarily with grant peer review, and advocates extensive reform. Sadly, the quest for excellence in research has not been accompanied by a quest for excellence in the evaluation of that excellence. To prevent further deterioration, it is proposed that conventional peer review be replaced by a new form of peer review, called "bicameral review".  In the sense that our justice system declares it better the guilty go free than that the innocent be condemned, we believe it better that poor research be supported than that excellent research be condemned to zero support. Donald Forsdyke

The Argument in a Nut-Shell Despite lip-service to the contrary, the grant agencies assess projects, not people. In the final analysis they hold it is better that a less able researcher carry out an approved project, than that a more able researcher carry out an unapproved project. Indeed, they hope with the funding carrot to coerce the more able researchers to carry out approved projects.     For the less able researchers this is not a problem. They just have to write an honest application stating what they want to do and why they want to do it. On the other hand, the more able researchers, who can see beyond the conventional wisdom, have serious difficulties. Grant writing is a marketing exercise that, more often than not, requires that that their "best" ideas be discarded, since, by definition, these ideas are difficult to understand and communicate (if not, the less able researchers would have already thought of them).      Thus, the more able researchers are tested, not on their abilities to come up with innovative ideas, but on their abilities to tune in to the conventional wisdom, and write an application with an appropriate degree of marketing spin. Many able researchers, and especially the most able, find this, not only distasteful, but impossible to do.     People find this difficult to understand. Why can't the researchers just write a simple application, and then, when they have the money, use it to do the work they want to do? Unfortunately (or fortunately, depending on your perspective) the most able researchers, although they come in all shapes and sizes, have one common attribute - integrity. They can no more discard this than a tortoise can discard its shell, or a giraffe its neck!        Thus, the major premise here is that peer review, as currently practiced in North America and many other places, is highly error-prone. It discriminates against the most able, so achieving the very opposite of what is desired. This means that over several decades peer-review has "dummed down" the Professoriat, decreased the quality of available "expert" advice, and impaired the process of scientific discovery.  What is the remedy?      Decision-making in uncertain environments (as any Wall Street analyst will tell you), should be guided by two cardinal rules: 1. Use the most objective parameters. 2. Hedge your bets (i.e. diversify). In the context of research funding this translates into: 1. Emphasize track record. 2. Avoid sharp cut-off points by using a sliding scale of fund allocation. There are four reasons why such obvious reforms have not been made: 1.Premise of impossibility not accepted. It is very difficult for those in Western culture systems to accept that some things are virtually impossible. If you have a problem, just appoint a committee of informed, well-intentioned, persons, and the best solution will emerge! When the resulting solution is attacked, the response is to shrug and declare no feasible alternative: "Like democracy it’s a terrible system, but it’s the best we have". 2.Winners do not want change. Those best placed to bring about reforms are the "winners" who have been supported by current peer review procedures. It is hard for them not to accept the syllogism: "I am an excellent researcher, the system recognizes that I am excellent, therefore the system must be excellent".   3. Losers think they see losers. It is difficult to win public support for reform because, admit it or not, the average member of the public is a "loser" (in the sense that only one person can be top of the class). All-to-readily we invent face-saving excuses, and all-to-readily we think we recognize the excuses of others. Why should the bleating, disenchanted researchers who are not funded be any different?   4. To compete needs big bucks. To keep up with top laboratories in other countries the "best" must be funded. This argument has much weight if one could really predict the "best". In practice, as funds get tighter, it translates into progressively moving the funding cut-off point higher. The ad absurdum argument is that ultimately there will be just one laboratory in a country with funds sufficient to compete with top laboratories elsewhere. This argument ignores the fact that there are many low-cost research projects where a country can compete very well. Somewhere along the line it has to be recognized that:   (i) great damage is inflicted at many levels by not funding more projects, and (ii) it may not be in the best interests of a country to put all its eggs in a few, very expensive, baskets.

http://post.queensu.ca/~forsdyke/peerrev.htm#The%20Argument%20in%20a%20Nut-Shell