By Professor D. P. Dash
Government ministries, research councils, universities and other agencies connected with research assess research proposals frequently. These agencies have an interest in allocating limited funds to research which has the potential to yield outcomes they consider relevant. Although one may assume that research projects that aim to produce immediate social or economic benefits should get priority in funding, the matter is seldom that simple.
First, there is no guarantee that a research project will achieve its objective. This is partly so because of the inherently uncertain nature of research. The funding agency must be satisfied that the research team has the necessary expertise and wherewithal to carry out the project successfully. Research projects may flounder due to poor project planning and management. For example, in health research, the required number of volunteers from a community may not be available easily.
Second, even if the research turns out to be successful, translating the results into practical benefits may not be straightforward. It may require further effort in terms of new infrastructure (e.g. a new type of communication network), technological retrofitting (e.g. modifying current machineries), or changing social behaviour (e.g. changing eating habits).
Third, research projects that do not aim at immediate social or economic value may also be attractive to funding agencies due to their potential to generate future developments in certain technical or professional fields. For example, the study of astronomy may trigger developments in satellite communication; study of a classical language may trigger developments in knowledge representation in artificial intelligence.
Finally, purely curiosity-driven basic research, even without envisaging any practical benefit, can still play a role in the development of knowledge. Such fundamental research, when published in reputed journals, can generate visibility and prestige for the researchers and the supporting agency.
Accordingly, there are different types of questions to be answered while assessing a research proposal: (a) Is it likely to succeed? (b) Can the results be translated to produce some practical benefit? (c) Is the project likely to trigger new developments in other fields? (d) Can the research generate prestige?
These questions may appear simple but are pretty difficult to answer in reality. First of all, these are all questions about the future. A good deal of subjectivity can enter into the assessment process. Research fund managers tend to look for evidence of past success on which they may base their assessment. This is applicable where a research group has a history of engagement in a specific area of research and there is a record of objective outcomes over time.
But, how does one assess new research groups that do not have such a track record? Well, the common approach is to look for at least one individual member in the group who has a positive track record.
In the world of research, there are various ways to assess the quality and quantity of contribution of an individual. “Bibliometric measures” are used for this purpose in many countries. Such measures focus on assessing the quality of research articles coming from a specific source (which could be an individual, an institution, a journal, etc).
In bibliometric analysis, the quality of an article is measured by the number of times other articles cite it. This logic is applied to compute a quality index for a whole set of articles (maybe written by one person, or originating from one institution, or published in one journal volume, etc). A common index is computed for a specific set of articles, such as the average number of citations to those articles (i.e. total number of citations to that set of articles divided by the number of articles in the set). This is called the “impact factor”, which has different forms and applications in research assessment.
Impact factors, based as they are on citation counts, reflect the level of peer recognition within the global research community. There has been considerable debate on the appropriate use of such measures. Sometimes an article is cited for the “wrong reasons”, for example, when a flaw is pointed out. Citation rates often have to do more with current academic fads than with the intrinsic merit of what is cited. There are phenomena such as “citation cartels” in which groups of authors mainly cite one another, ignoring other relevant work.
The validity of such bibliometric research assessment has been a hot topic of debate. A recent opinion survey of a broad sample of scientists from all over the world showed that the opinion is “neither positive nor negative”. Despite its shortcomings, there are practical benefits of having a bibliometric measure of research quality. For instance, in institutional environments where the expertise to assess research quality is not available, a bibliometric measure does provide a reasonable indication. Of course, it is not perfect.
Professor D. P. Dash is Head of the School of Business and Design at Swinburne University of Technology Sarawak Campus. He can be contacted at ddash@swinburne.edu.my