Responses to this article in BMJ, 6 December 1986
"Research grants: problems and options"
Brian Martin's publications on science and society
Brian Martin's publications
Brian Martin's website
In many branches of science research grants play a key part in funding researchers and projects. In English speaking countries a fairly high proportion of research money is administered through grant systems. Yet despite the importance of research grants there has been little study of biases affecting the grant giving process.
A key mechanism in most grant systems is peer review, which has been the focus of increasing attention in recent years.[1,2] A major difficulty in studying biases in peer review is the anonymity of reviewers. Recently, freedom of information legislation in Australia was used to expose a case of apparent bias in the denial of a research grant proposal, thus illustrating a number of important aspects of the general problem of bias.
In Australia the National Health and Medical Research Council (NHMRC) is the largest grant giving body for the medical sciences. Administered through the Commonwealth government's Department of Health, in 1986 it dispensed 32 million Australian dollars to roughly 750 projects.
Recently there has been limited publicity about the case of an unsuccessful applicant to the NHMRC.[3,4] The account here is based on documents about the case made available to me by the scientist in question, who prefers to remain anonymous and is referred to here as Dr Smith.
Dr Smith applied for NHMRC grants on three occasions, in 1976, 1979, and 1982; only the 1979 application was successful. Dr Smith had over 30 years' experience in human development and behaviour and family functioning, with many publications in international journals and high professional standing both in Australia and overseas. After the rejection of the 1976 application Dr Smith wrote to the NHMRC secretary asking for reasons. In November the secretary replied that the reports of assessors had to remain confidential but, nevertheless, he could pass on a few general comments based on the reports. Dr Smith was very unhappy with this and immediately sent a lengthy letter to the NHMRC secretary defending the application. There was no reply from the NHMRC. After considerable delay Dr Smith undertook the research, making financial sacrifices to do so.
This might have been the end of the matter, except for an important development: the introduction in 1982 of freedom of information legislation in Australia. The federal legislation covered the Department of Health. After the rejection of the 1982 application it seemed opportune for Dr Smith to request information about all the grant applications.
In 1976, apparently the standard procedure for grants was for NHMRC applications to be sent to several external assessors, either in Australia or overseas, who were experts in the applicant's subject. The assessors were expected to comment on the application and rate it by ticking one of six boxes graded 1 (poor) to 6 (outstanding). The application was assigned to a particular member of a relevant committee of the NHMRC. This member, called the spokesman, reported to the committee on his own personal rating as well as those of the external assessors. Then the other members of the committee wrote down their ratings, and the average of the ratings by the committee members was used to decide whether to offer a grant. Dr Smith sought documents through the freedom of information legislation on all these facets of the 1976, 1979, and 1982 grant applications.
The department of health initially provided copies of the reports submitted by the external assessors in each of the three years and by the interviewing committees responsible for the final grant recommendations. The ratings that went into these reports were deleted. Dr Smith requested an internal review of the decision to delete the ratings. This led to the release of the ratings of the 1976 and 1979 assessors. The ratings of the 1982 assessor were withheld on the grounds of "breach of confidentiality." Access to the final ratings of the 1976, 1979, and 1982 interviewing committees was also denied by the department of health as being "contrary to the public interest" because "the disclosure of these ratings would prejudice the effectiveness of the 'peer review process' by which medical research funds are allocated."
This decision was not surprising as it is common for government departments to reveal as little as possible on sensitive issues in response to freedom of information requests. The standard response to this obstruction is to appeal against the department's refusal. In several cases departments have withdrawn their cases and supplied the requested documents just before the appeal was to be heard. In the Smith case an administrative appeals tribunal finally heard the case and ruled in favour of Dr Smith. This was a pivotal decision as it potentially opened up the federal grant giving process to much more scrutiny.
In Dr Smith's 1976 application there were two external assessors. One supported the project, giving it a rating of 5 (very good). The second assessor offered a variety of comments, both supportive and critical, and proposed providing support for a pilot project (funding for one or two years). This assessor did not include a numerical rating. By contrast, the spokesman's interview notes (a standard form) were almost uniformly negative. Most critically, the independent assessors' priority ratings are written down as "1 and 'not entered in a box'." The final committee rating was 1, the lowest possible.
It seems reasonable to infer that the spokesman misrepresented the assessors' reports to the committee. One assessor's rating was altered from 5 to 1, and the other became "not entered in a box" with no mention of the recommendation for a pilot project. The spokesman seems to have influenced the rest of the committee to give the project a rating of 1.
Although it seems certain that an injustice was perpetrated, there is no way to prove bias. Nevertheless, it is worth spelling out some possible reasons for the spokesman's hostility to the project. Firstly, Dr Smith is a strong, confident, and outspoken woman scientist in a male dominated research area. As such, she is likely to be seen as a threat by some male scientists. Secondly, Dr Smith approached problems in her subject in quite a different fashion from the spokesman. Dr Smith used direct detailed observation with a small sample to analyse nonverbal processes, whereas the spokesman built his career on gathering data through questionnaires. Thus there may have been elements of a paradigms conflict between them. Finally, Dr Smith had a PhD, and the spokesman had an MD. Dr Smith had occasionally appeared at meetings of MD colleagues and disagreed with them. The PhD versus MD issue within the medical research community symbolises a number of differences that are often keenly felt.
In short, Dr Smith was an outspoken woman with a PhD doing the wrong sort of research so far as the spokesman was concerned. This may explain his hostility. But in this case at least it is impossible to know the subjective origins of bias. All that the documents show is the altering of one of the assessor's ratings and a consistently negative report at odds with the expressed views of the assessors.
Dr Smith cared less about the failure of her 1976 grant application, as her research had then proceeded satisfactorily anyway, than about the falsification of the assessors' reports and the bias introduced into the NHMRC peer review system. After all, there is little point in knowing about past injustice if nothing is done to make those responsible accountable for their actions.
Accordingly, Dr Smith lodged a complaint against the NHMRC with the Commonwealth ombudsman. The outcome in relation to the 1976 grant application was not heartening to Dr Smith. For example, the spokesman's act of registering the 5 rating of one assessor as a 1 was referred to by the ombudsman as an "error." No judgment was made by the ombudsman about whether the spokesman was likely to have misled the other members of the committee. Concerning the letter sent to Dr Smith by the secretary of the NHMRC (telling her in general about the assessors' comments but not of their positive recommendations and omitting any reference to the spokesman), the ombudsman said that the secretary may have "encountered a drafting problem" in telling Dr Smith the reasons for the failure of her application without revealing the assessors' reports. Suffice it to say that the ombudsman took a very cautious approach to the NHMRC. Although acknowledging multiple deficiencies in the granting system, such as lack of proper documentation of certain aspects of decision making, the ombudsman examined nearly every point in a way which exonerated the NHMRC from any responsibility.
There are many other facets to the Smith case, including the intricacies of how she obtained various documents through the freedom of information system (and through other channels) and her grievances concerning the unsuccessful 1982 NHMRC application. Yet even the basic outline of the 1976 grant case provides some instructive lessons.
Grant giving bodies are important parts of the social system of science, but their operations are largely carried out in secret by elite members of the scientific community. The very limited available evidence suggests that various types of bias occur in the allocation of grants. Informally, many scientists believe that they must "play it safe" in grant applications because of discrimination against unorthodox ideas and projects that might offend those with vested interests.
The referees of grant applications are usually leading figures in their subject; as such they are almost always exponents of the prevailing conceptual structures, methods, and orientations in their subject, or in other words representatives of the current paradigm.
Grant proposals that deal with what are considered to be unorthodox ideas, therefore, have a greatly reduced chance of success. Lynn Margulis, for example, describes her experience in applying for National Science Foundation grants concerning her work on a possible endosymbiotic origin for the microtubule system:
I was told by an NSF [National Science Foundation] grants officer (after having been supported nicely for several years) that 'important' scientists did not like the theory presented in a book I had written and that they would never fund my work. I was actually told that I should never apply again to the cell biology group at NSF.
Because it is widely recognised by scientists that unorthodox ideas have little chance of receiving funding, proposals are commonly self censored.
David Horrobin gives the classic account of a different source of bias in grant refereeing: ignorance and incompetence. Drawing from his own experience in applying for grants to study the action of prolactin in sheep and humans, he describes how some referees, including some "highly respected ones in top academic positions," were ignorant of current work on the subject and apparently had not fully read (or comprehended) the grant application. Many scientists have their own stories about incompetent referees' reports.
One explanation for the poor performance of some referees is that they are overworked and fail to keep up with what is happening in their subject. The leading people in many specialties are research administrators rather than research workers and so may be quite out of touch with research developments. The secrecy that shrouds most refereeing prevents accountability and enables some scientists to get by with shoddy efforts that would be an acute embarrassment if disclosed publicly.
There is substantial documentation of the role of vested interests - such as governments, large corporations, and professions - in suppressing research threatening to them - for example, by blocking appointments, smearing reputations, blocking promotions and publications, and sacking people. Preventing potential critics from gaining research grants is one such method of suppression. Because of the secrecy smothering most grant giving operations, the number of documented cases of suppression is small. Aside from the case of Dr Smith, there seem to be only two other published cases of alleged prejudice in awarding research grants in Australia.[10,11]
A well known case in the United States is that of Dr Thomas Mancuso, who received funding for many years from the United States Department of Energy to study the effects of low level ionising radiation on workers at the Hanford nuclear reprocessing plant in the state of Washington. The attack on Mancuso was stimulated by his refusal to issue a press release rebutting the finding by another researcher, who found an increased risk of cancer among Hanford workers. Mancuso refused because he had not finished his study. This action worried key people in the Department of Energy who were afraid that Mancuso might obtain and publicise similar results. Irwin Bross describes what happened next:
The DOE [Department of Energy] called in six reviewers to 'decide' whether or not to continue support for Dr Mancuso at the University of Pittsburg. Four of the six recommended that the support be continued, two reviews were unfavorable.... on January 8, 1976, a DOE staffer (who subsequently left for the private sector) produced a memo recommending termination of the Mancuso contract. The recommendation was actually carried out by a second DOE staffer who replaced him. As 'justification,' the DOE memo cited the two unfavorable reviews, only one of which had recommended termination (and transfer to another school of public health). The four favorable reviews were not mentioned.... the contract for the Hanford study that had been taken away from Dr Mancuso had been transferred to Battelle West, a private contractor. There, it was under the administrative control of the ex-DOE staffer who had written the termination memo.
Little of this information would ever have been revealed except for a congressional investigation and the use of the Freedom of Information Act.
The allegation that vested interests bias decisions about grants is a strong one and very difficult to prove. It is important to emphasise that subjective bias is not necessary. Most referees and research administrators are well meaning and convinced that they have the best reasons for their decisions. The key issue is whether the actual decisions would have been different had the vested interests not existed. The conceptions held by many scientists about what is good, interesting research are affected by payoffs in potential applications, job prospects, and future research opportunities. Once these conceptions are influenced by vested interests bias in grant giving follows.
One corrective to the most blatant abuses in the grant system is to open the assessment process to full professional and public scrutiny, removing the protection of anonymity. In the Smith case the previously confidential documents showed errors or deceptions that would not have escaped attention had documents been made available to the applicant at the time.
It will not be easy to challenge the secrecy of most grant giving processes. The power of many scientific elites is built on and exercised through the allocation of grants; opening the system to outside scrutiny would be opposed by present and aspiring elites on whatever grounds they could muster. Yet any disadvantages of removing anonymity must be weighed against the injustices that are made possible by the present system, including wasted scientific effort and pressures towards conformity.
Peer review sounds worthwhile, but it has no necessary link with anonymity. If anonymous peer review were called instead "unaccountable evaluation with occasional backstabbing" it would be less defendable. On the other hand, removing anonymity will not magically remove all abuses. The difficulties faced by Dr Smith in demanding accountability from the NHMRC are instructive. Scientists routinely claim that their expertise gives them the right to make judgments that non-specialists are incapable of understanding, and it is possible to imagine an open grant giving system in which jargon, credentials, and complex procedures are used to mask judgments made on non-scientific grounds.
My aim here is not to propose a replacement for the grant system and peer review but rather to point to the need for a much wider discussion of the goals and methods of the present system and of possible reforms and alternatives. This discussion should encompass not only administrators and scientists but also members of the general public, all of whom have a stake in fairness and the promotion of scholarship in service to the community.
Valuable comments were received from Richard Davis, Clyde Manwell, and Dr Smith.
1 Harnad S, ed. Peer commentary on peer review. Cambridge: Cambridge University Press, 1983.
2 Lock S. A difficult balance: editorial peer review in medicine. London: Nuffield Provincial Hospitals Trust, 1985.
3 Waterford J. FOI ruling opens up grants system. Canberra Times 1985 January 20:1.
4 McIntosh P, Koval R. Blewett asked to check on research grant processes. Age (Melbourne) 1985 July 3:17.
5 Waterford J. A layman's guide to making an appeal. Rupert Public Interest Journal 1985 March;No 12:15-7.
6 Kuhn TS. The structure of scientific revolutions. Chicago: University of Chicago Press, 1970.
7 Margulis L. Peer review attacked (letter). The Sciences 1977; 17:5,31.
8 Horrobin DF. Referees and research administrators: barriers to scientific research? Br Med J 1974,2:216-8.
9 Martin B, Baker CMA, Manwell C, Pugh C, eds. Intellectual suppression. Sydney: Angus and Robertson, 1986.
10 Davis R. Peer review and the Australian Research Grants Committee: a cautionary tale. In: Martin B, Baker CMA, Manwell C, Pugh C, eds. Intellectual suppression. Sydney: Angus and Robertson, 1986: 50-5.
11 Manwell C. Peer review: a case history from the Australian Research Grants Committee. Search 1979; 10:81-6.
12 Bross IDJ. Scientific strategies to save your life. New York: Marcel Dekker, 1981:219.
13 Davis R. Anonymity: the cancer of academia. Education research and perspectives 1979; 6:3-11.
(Accepted 10 June 1986)
Department of History and Philosophy of Science, University of Wollongong, Wollongong, New South Wales, 2500 Australia
BRIAN MARTIN, PHD, lecturer