Institute of Gender and Health: Stakeholder Engagement Report to the CIHR Peer Review Expert Panel

November 2016

Main Messages

  1. Funding equity. Significant concerns were raised about: 1) how new investigators are evaluated and how they are often held to the same standards as senior investigators, 2) how there appears to be a significant bias in funding towards the biomedical sciences, and 3) the poor success rate of senior and mid-career women researchers for the Foundation Scheme Grants.
  2. Lack of reviewer expertise, especially for sex and gender considerations. Reviewers are often poorly matched with applicants in terms of required expertise. There were also concerns that reviewers might not have the required expertise to properly assess the integration of sex and gender considerations within research protocols. Many stakeholders also believed that the reviewer pool should be increased to include non-CIHR grant holders to expand the number of potential reviewers with expertise in a given health research area.
  3. Suggested improvements to the peer review process. Stakeholders were troubled by the fact that most reviewers do not read the content of the question box in the administrative section, which asks applicants to describe how they account for sex and/or gender within their research proposal. Specific instructions should be given to reviewers on where to find the box because applicants often include highly relevant background and methodological information that cannot be found elsewhere in the proposal. Several stakeholders also supported the idea that applicants should be given the opportunity to respond to reviewer comments.

Stakeholder Engagement Approach

On November 10th 2016, IGH held a focus group with twenty members of the IGH community and asked for their input on the design and adjudication processes of CIHR's investigator-initiated programs in relation to the CIHR mandate, using the PREP questions as a guide.

Participants

The majority of the focus group participants were either applicants or reviewers for the last round of investigator-initiated CIHR project and foundation scheme funding opportunities. There were 25 researchers in attendance including three trainees.

There was some confusion among the group as to whether they should report their experience in the last competition, or comment on the changes being implemented for future competitions. The group decided to base their responses on their past lived experiences.

Summary of Stakeholder Input

Question 1: Does the design of CIHR's reforms of investigator-initiated programs and peer review processes address their original objectives?

The overwhelming consensus was that the CIHR's reforms of investigator-initiated programs and peer review processes did not meet their original objectives. Specific concerns are highlighted below.

  • Concerns were raised about how new investigators are evaluated and how they are often held to the same standards as senior investigators. It was suggested that peer reviewers be taught how to review new investigators, and that new investigators would benefit from being evaluated separately.
  • It was generally agreed that the overall quality of the reviews diminished following the reforms. The assignment of reviewers seemed "rather random". Reviewers were often poorly matched with applicants in terms of required expertise. The goal to achieve scientific excellence was not attained.
  • The face-to-face meetings must come back. They facilitate discussion, allow a better comparison of applications, more transparently permit consensus-reaching and, perhaps most importantly, ensure reviewer accountability. Reviewers put more effort into evaluating a grant knowing they will have to read their evaluation to other peers.
  • In terms of the future process for reviewer selection, based on holding a CIHR grant, it was highlighted that large pools of experts exist outside the CIHR sphere and could be solicited for review. With regards to sex and gender considerations, for example, many experts can be found within the Social Sciences and Humanities Research community. If reviewer selection is limited to CIHR grant awardees, a large pool of reviewer expertise may be missed. Many felt that that the pool of expertise is "outside" of CIHR because they have been relegated there, and that there is work to be to address the void that has been created by the structural schism between CIHR and SSHRC. The latter has turned away from health while the former has done little to accommodate social scientists.  One study was cited that discusses this issue: https://ejournals.library.ualberta.ca/index.php/CJS/article/view/21329/17200
  • The majority of stakeholders perceived that there seems to be a strong and significant bias towards the biomedical sciences, indicating that researchers whose work falls under CIHR pillars III and IV have poorer funding success rates. Similarly, the social sciences are poorly represented in the successful grant applications.
  • There was also general concern about the success rate of women for the foundation scheme grants. Senior- and mid-careers female researchers were funded at a rate that was half the rate of male researchers.
  • Researchers who hold Foundation grants were very critical of the fact that co-PIs could not be named on the grant. This is a significant issue considering that the Foundation grant replaced previous operating grants that included co-PIs, who now no longer have access to these funds.
Question 2: Do the changes in program architecture and peer review allow CIHR to address the challenges posed by the breadth of its mandate, the evolving nature of science, and the growth of interdisciplinary research?

Several concerns were raised regarding the use of the question box in the administrative section asking applicants to describe how they are accounting for sex and/or gender within their research proposal, or to provide an explanation for why they chose not to. Part of CIHR's mandate is to respond to the Government's Sex- and Gender-Based Analysis Policy for all Health Portfolio partners. The policy stipulates that all research, programs and policies funded by government should meet the diverse needs of men and women. Interpretation from CIHR's point of view is that research funded by CIHR should apply to and benefit the health of men and women in an equitable fashion.

  • Many stakeholders were upset that most reviewers do not read this section and do not provide feedback to the applicants regarding the information provided in the box. In comparison, the National Institutes of Health (NIH) in the United States has a section within the proposal about the inclusion of women, minorities and consideration of sex as a biological variable, which gets evaluated by the peer-review committee.
  • Since the box is placed in the administrative section of the grant proposals, most reviewers do not know where to look. Specific instructions should be given to reviewers on where to find the box because applicants often write highly relevant background and methodological information that cannot be found elsewhere in the proposal. Alternatively, it was suggested that the content of the box could be moved to the proposal section for ease of accessibility.
  • There were also concerns that reviewers might not have the required expertise to properly assess the integration of sex and gender considerations within research protocols. It was suggested that reviewers take at least one of the three IGH-developed online competency modules designed to help peer-reviewers assess the integration of sex and gender considerations within research protocols and publications. This would, at the very least, help increase reviewers' perceived self-efficacy.
  • Another issue raised was that, while most grant applications fall into CIHR research pillars I and II, the majority of sex and gender experts conduct research that fall into pillars III and IV.
  • Stakeholders also highlighted the need to provide a section for applications where they can provide evidence for multi-/inter-disciplinarity. While this can be partly done by looking at the co-investigators of a grant, it really never tells the whole story with regards the various partnerships within and outside academia and their respective roles in the proposed research projects.
Question 3: What challenges in adjudication of applications for funding have been identified by public funding agencies internationally and in the literature on peer review and how do CIHR's reforms address these?

  • It was proposed that each peer-review committee should have at least one expert on sex and gender to assist the committee with comparative assessments of sex and gender integration within the proposals.
  • There was also a general consensus that we need to increase basic competence for the assessment of sex and gender considerations in addition to using experts. Requiring completion of an online training module on sex and gender (www.discoversexandgender.ca) would achieve this goal. Raising the level of awareness of how to include sex and gender in the methods of the proposal will lead to better health research and health outcomes for Canadians.
  • The chairs and scientific officers of the peer-review committees should complete the online course and ensure that sex and gender considerations are properly addressed, when applicable.
  • Several CIHR peer-reviewers highlighted that in the European Union, sex and gender are often part of the evaluation criteria in open investigator-initiated funding competitions. This should be considered for the evaluation criteria for CIHR open competitions when applicable.
Question 4: Are the mechanisms set up by CIHR, including but not limited to the College of Reviewers, appropriate and sufficient to ensure peer review quality and impacts?

While addressing this question, several of members of our community wished to express concerns that extended beyond the open competitions. Although some recommendations do apply to the open competitions, others more specifically applied to institute-led strategic initiatives.

  • It was agreed that applicants should be required to integrate sex and/or gender or adequately explain why they do not apply to their research. Too many get away with not providing any information in the box by simply writing "N/A". This sort of requirement is currently enforced at the National Institutes for Health (NIH) in the United States, and aligns with the principles of high quality peer review.
  • Some stakeholders wanted to highlight that, in strategic funding opportunities, there is often a disconnect between the conceptualization of a funding opportunity and the review process. Reviewers are often not sufficiently instructed on the most important evaluation criteria or provided with enough background material to properly review the applications with the intended objectives of the funding opportunity in mind.
  • Sex and gender capacity building should be linked to all funding opportunities (not just IGH-funded calls). For example, direct links to resources such as the IGH training modules should be made explicitly available to applicants on all funding opportunities including the open competitions. This will help mainstream awareness of the prominent roles played by sex and gender in contributing to health and disease, and increase the quality of peer review with respect to sex and gender science.
  • In strategic funding competitions, the roles and responsibilities of sex and gender champions must be better defined from the onset. Mechanisms need to be put in place to ensure sex and gender champions are not last-minute token additions; they need to be involved throughout the whole research process, including knowledge translation and implementation activities.
  • Concerns were raised about the evaluation criteria used in the Foundation scheme competition. It was felt that the leadership and productivity criteria (e.g. the H Index) unfairly advantaged biomedical researchers.
Question 5: What are international best practices in peer review that should be considered by CIHR to enhance quality and efficiency of its systems?

  • In the United Kingdom, citizens are often included in the adjudication committee to be involved in the discussion on the potential impact of research findings. This ensures that, no matter how good the science may be in a given application, the likelihood of the application receiving funding is reduced if it does not have a significant potential impact for society.
  • Several stakeholders supported the idea that applicants should be given the opportunity to respond to reviewer comments. One interesting model is a two-stage funding model currently being used in Australia, where a portion of the funds is awarded only after review of the applicant responses to the initial review. The main drawback of this model is that insufficient space is provided to properly respond to all of the comments. While the two-stage review process might increase reviewer burden, it could lead to better science. The CIHR Institute of Population and Public Health has also experimented with this model.
  • The National Institutes of Health (NIH) in the United States has a section within the proposal about the inclusion of women and minorities to ensure that the science applies to these populations. The content of this section gets evaluated by NIH peer-review committees. The approach is very similar to what is being proposed by members of our community: that CIHR peer reviewers be instructed to evaluate the contents of the sex and gender box currently located in the administrative section of the proposal. 
Question 6: What are the leading indicators and methods through which CIHR could evaluate the quality and efficiency of its peer review systems going forward?

The surveyed stakeholders identified several key indicators that could be used to evaluate the quality and efficiency of CIHR's peer review system.

  • Track the number of women funded (and grant success rates) to ensure equity.
  • Track the number of reviewers that decline to review in relation to their gender, discipline, pillar, etc.
  • Track the reasons given by researchers when they decline to perform CIHR peer review duties.
  • Track the number of grants where sex and gender are appropriately included in the research question.
  • Track grants for a variety of subtopics to identify if certain areas of research have disproportionally high or low success rates. This could help identify certain biases in the peer-review system.
  • Track and compare the grant success rates for each CIHR research pillar to achieve a better balance across the pillars and to ensure equity.
  • Ask reviewers to fill out a short survey on how to improve the efficiency and quality of the review process.
  • Track the number of peer-review committees that have a sex- and gender-based analysis champion/expert.
  • Track the number of scientific officers, chairs, reviewers and applicants who can show proof of completion of the online sex and gender training modules (a certificate of completion is provided).
Date modified: