Institute of Population and Public Health: Stakeholder Engagement Report to the CIHR Peer Review Expert Panel

November 2016

Main Messages

1. According to the population and public health (PPH) research community, the complexity, heterogeneity and multi-component nature of PPH research (i.e. pillar 4) requires careful attention in the recruitment and training of peer reviewers and in the matching of applications to reviewers. First, the unit of analysis in PPH research is populations, which is different from individually-oriented studies. Second, PPH is not the same as precision or personalized medicine. These and other kinds of distinctions must be made clear to reviewers. To ensure a robust and appropriate pool of reviewers, they also recommend that recruitment efforts undertaken by the College of Reviewers be extended to include international pillar 4 reviewers while ensuring adequate training is offered to promote better understanding of the Canadian health research, policy and practice context.
2. The PPH research community feels that CIHR's rules and broad health research mandate are not currently being consistently respected by peer reviewers. CIHR must ensure that reviewer comments are not admissible if they reflect a misunderstanding of CIHR support for pillar 4 research. IPPH stakeholders recommended the following: First, better orientation about the breadth of health research funded by CIHR (including pillar 4) must be provided for all reviewers with ongoing feedback on the appropriateness of reviews. Second, the multi-component, complex and heterogeneous nature of PPH grant applications requires the integration of face-to-face interactions as part of the adjudication process in order to keep the level of competency in 'check' while ensuring that no one reviewer overstates their level of competency on any section of a pillar 4 research grant.
3. The PPH research community calls for transparent and ongoing monitoring of application pressure and success rates by pillar of research (including pillar 4) as part of project and foundation scheme implementation plans. They further request that pillar 4 sensitive indicators that provide early indication of systemic biases affecting the quantity, quality and diversity of PPH research funded be incorporated in future performance measurement and evaluation efforts.

Stakeholder Engagement Approach

The Institute of Population and Public Health (IPPH) conducted two targeted focus groups by teleconference on October 11th (group 1) and October 28th, 2016 (group 2). An initial invitation from IPPH's Scientific Director was issued by email to current and past applied public health chairholders. Two follow-up reminders were sent to Chairs. Chairs were also encouraged to invite one population and public health (PPH) colleague who could offer a different perspective from their own (e.g.,'trainee, junior faculty member, senior researcher, very'successful with CIHR grants, not successful with CIHR grants, focused locally or globally, etc.). Input on the six questions for the review was also solicited in writing from those who could not attend either focus group. Feedback specific to PPH research (i.e. pillar 4) was particularly encouraged. In some instances, specific questions were asked of participants. These are noted in the summary and the focus group is specified.

Participants

Current and past applied public health Chairholders (n = 26) funded by IPPH and other partners such as the Public Health Agency of Canada were invited to participate because they are a representative segment of the broader PPH research community. Current/past chairs are mid-career (6-12 years of experience after their last qualifying degree) and senior investigators in PPH research at the Associate or Full Professor level. They represent a variety of disciplines and fields (e.g. nursing, community medicine, geography, health promotion, etc.) and are based in academic institutions across Canada. Through this program, Chairs are/were funded to implement applied programs of research, capacity building, and knowledge translation on population health interventions (i.e. study of policies and programs that have health and health equity impacts at the population level). A total of nine Chairs participated in two one-hour teleconferences co-facilitated by Steven Hoffman and Erica Di Ruggiero. The feedback from Chairs is representative of views shared by the broader PPH research community about the reforms and peer review of pillar 4 applications (i.e. sources include: visits to schools of public health conducted by the former IPPH Scientific Director (SD), the Fall 2016 listening tour conducted by the current IPPH SD and the IPPH Pillar 4 Forum Report)

Summary of Stakeholder Input

Question 1: Does the design of CIHR's reforms of investigator-initiated programs and peer review processes address their original objectives?

Objectives not clear and not met

Participants reported an overall lack of clarity about the objectives of the reforms and what "problems" the reforms were intended to address and how they would benefit pillar 4 researchers. After further explanation by IPPH representatives, the following feedback was obtained (further captured in greater detail under questions #2 and #3).

The reforms were characterized by some participants as less than satisfying. Participants noted that objectives including the need for the reforms to be responsive to all pillars of health research (including pillar 4) were notconsidered as met. Participants expressed a great deal of uncertainty about what to expect in the future based on recent experience from a pillar perspective, and in contrast to experience under the previous investigator-initiated program. Most of the concern from participants was about the new peer review system and whether it was adequate to assess the complexity of pillar 4 research.

Degree of Fit with pillar 4 research

Some participants indicated that they didn't apply to the foundation scheme given the "widespread perception" that the scheme is not geared to support PPH (pillar 4) research and that reviewers assigned don't understand PPH research. Participants further noted that the foundation scheme doesn't seem to favour innovative research.

Transparency by pillar

There was also interest from participants in knowing how the PPH research community has and will continue to fare in the foundation and project schemes, recognizing that pillar 4 researchers have historically been known to be quite harsh in their reviews, unduly focusing on methods. Recent results in the project scheme competition were not considered encouraging for pillar 4 research. Participants called on CIHR to be transparent about success rates by pillar in the foundation and project schemes.

Question 2: Do the changes in program architecture and peer review allow CIHR to address the challenges posed by the breadth of its mandate, the evolving nature of science, and the growth of interdisciplinary research?

According to participants, the state of what is more broadly happening in research in Canada needs to be taken into account. They view the manner in which pillar 4 researchers conduct research could be influential with success rates of 20% or 21%, but with 12%, it is not possible. They further noted that it will be difficult to exercise the level of influence through PPH research especially at a time when there is even greater capacity and interest in PPH evidence.

In essence, participants feel that what has been put into place is an increase in the number of people reviewing each grant but the development of the consensus that occurs in face-to-face meetings is not coming across in the scores for pillar 4 applications. It is their view that rankings force a "do or die" decision in terms of money – CIHR needs to acknowledge and systematically deal with this reality, especially if research funding levels don't increase in the near future.

Question 3: What challenges in adjudication of applications for funding have been identified by public funding agencies internationally and in the literature on peer review and how do CIHR's reforms address these?

Misunderstanding of CIHR rules and broad pillar mandate

Participants felt that CIHR's rules and broad mandate are not currently being respected by peer reviewers. They recommend that reviewer comments not be admissible in final reviews if they reflect a misunderstanding of CIHR support for PPH research.

Quality and appropriateness of reviews

According to participants, the expertise in PPH was "abysmal" (ref. to last project scheme competition). The comments from reviewers exemplified a clear lack of understanding of PPH research. Based on the experiences of some participants, applications seemed to have been reviewed by basic scientists rather than experts in PPH. Terms such as "remote" and "random" expertise and "unhelpful" comments were used to characterize such reviews. Only one participant mentioned that in the last round for the project scheme, reviewers seemed well-matched and feedback was topic-specific and methodologically cogent for pillar 4 applications. Participants called for greater consistency between reviewers and consistency in reviews between grants.

Pool of reviewers

Some participants questioned why researchers are excluded from reviewing if they are the PI on another grant. It is their view that the pool is already very small for people who can review, and review well, especially for PPH grants. They further questioned whether CIHR will have a pool large enough to support reviewing with such restrictions in place.

Value of Face-to-Face

From a pillar 4 perspective, participants felt that the multi-component, complex and heterogeneous nature of PPH grant applications requires the integration of face-to-face (FTF) interactions. They felt that the FTF panel system was a way to keep the level of competency in 'check' while ensuring that no one reviewer overstated their level of competency on any section of a pillar 4 grant. According to participants, FTF interactions force reviewers to do a better job reviewing through peer pressure at the meeting. The number of reviewers (i.e. 5) per grant was not seen by participants as a solution to reviewing pillar 4 grants. It was noted that five people will have different readings of a proposal while not necessarily understanding the whole proposal. Reviewers should be able to rate the section(s) of the pillar 4 proposal, but avoid rating sections where they have superficial understanding of the content.

Promote better understanding of PPH research

To ensure accurate review, participants recommended that reviewers have a clear understanding of PPH and how it is different from other fields. The fact that there are more reviewers is positive, but if reviewers do not understand what the grant is primarily about, it creates a lot of noise in the review, which is problematic according to participants. Participants recommended that CIHR help reviewers better understand the breadth of PPH research, which in part, also creates demands of applicants to be more explicit in their grants.

Participants reported that PPH research is complex and heterogeneous drawing on many disciplines, theories, methods and fields of research. It is their view that this heterogeneity can create challenges in the adjudication process. Orientation, reinforcement, and ongoing feedback should be given high priority by CIHR. Participants recommended training on the vocabulary used in PPH research. For instance, the unit of analysis in PPH is populations, which is different from individually-oriented studies. Participants further noted that PPH is not the same as precision or personalized medicine. They felt that these and other kinds of distinctions need to be made clear to reviewers.

Participants commented on the opportunity of pillar 4 Research to reach over to other groups and fields (e.g. public policy, study of complex health interventions (CHI)). Their view was not only about how PPH is different, but how it reaches over and complements over fields (e.g. CHI are relevant to clinical research). Participants recommended orientation of reviewers to facilitate greater understanding of these distinctions and complementarities between pillar 4 and other research fields and that these can co-exist within interdisciplinary, cross-pillar grant applications.

Question 4: Are the mechanisms set up by CIHR, including but not limited to the College of Reviewers, appropriate and sufficient to ensure peer review quality and impacts?

Participants felt that CIHR has taken too long to get the College of Reviewers off the ground. They also questioned how successful the recruitment from CIHR had been so far. For example, they asked whether CIHR has the numbers they were aiming for recruitment, considering CIHR appears to have reached the end of the projected process. Greater and more targeted outreach to recruit or encourage other colleagues in PPH research to join the College is also needed, according to participants.

When asked, there was no consensus among participants on whether peer review should be mandatory for those who hold CIHR grants as a way of giving back (group 2; n = 6). One of the potential 'cons' reported by pillar 4 participants is that if they are forced, they may not do as good a job reviewing as others who have volunteered.

Capacity building for reviewers

When asked what can be done to improve the match between peer reviewers and pillar 4 proposals, it was suggested that more time and care be invested in training and socializing reviewers to their role. They felt that reviewers don't just need modules and teleconferences but also a summer school on peer reviewing. Though resource intensive, they noted that it can help reviewers understand how to handle being out of their depth with some proposals they may be assigned. Participants felt that this was particularly important given the heterogeneity of PPH research (see question #3).

International reviewers

When asked if CIHR should approach international reviewers (group 2; n = 6), all felt it was a good idea for pillar 4 research but some caution was expressed. Participants commented that the pool of PPH researchers is smaller than for other fields (e.g. biomedical). However, they further noted that pillar 4 research is very context-dependent and so those who don't understand the Canadian context might miss the point in the review process. When participants were asked the question again, there was no strong opposition to involving international reviewers from a pillar 4 perspective. It was felt that some applications might be more appropriate for international reviews to assess. According to participants, it would be more beneficial in the long run if CIHR developed a cadre of international specialists who could review regularly for CIHR to complement and enhance the existing pool of Canadian pillar 4 researchers.

Feedback loop for reviewers

Participants emphasized the importance of provide feedback to pillar 4 reviewers on the quality of their reviews in order to improve the effectiveness of future reviews. They suggested that quality scores for reviews be assigned. They queried whether pillar sensitive criteria would allow the College of Reviewers to score the reviews and make recommendations about who should remain in the College and be assigned to pillar 4 applications.

Complementary mechanisms

The use of an internal peer review system within academic institutions was touted by participants as an excellent strategy to improve the quality of pillar 4 applications before they are submitted. Based on this particular institution's experience, this strategy has demonstrably increased the collective success of pillar 4 researchers in CIHR competitions. It has also reinforced the need to better understand the breadth of research in PPH.

Question 5: What are international best practices in peer review that should be considered by CIHR to enhance quality and efficiency of its systems?

All participants had reviewed for other granting councils within and outside of Canada. The Heart and Stroke Foundation (HSF) and the European Commission were identified by some as funders to learn from (group 1), while others (group 2) considered these organizations to have pretty traditional processes, much like CIHR used to be. One participant mentioned that the NIH has a system where they give feedback to those classified the lowest in an effort to strengthen future submissions.

When asked to contrast their experience with CIHR reviews in open vs. strategic competitions, one participant indicated that the reviews were better in the open, whereas the rest of participants indicated having the opposite experience (group 2; n = 6). It was recommended that CIHR learn more systematically from its own peer review experience across open and strategic competitions.

Question 6: What are the leading indicators and methods through which CIHR could evaluate the quality and efficiency of its peer review systems going forward?

One participant highlighted the need to differentiate between efficiency and burden, which are two different constructs. Efficiency is output focused, while burden is resource heavy. Efficiency doesn't necessarily mean quality. If efficiency is the key, quality will take a hit. If quality is the key, then the reviews will inherently be a bit burdensome. While this is not unique to pillar 4, participants felt that the quantity and quality of pillar 4 research funded would be affected if the right balance between these constructs wasn't achieved.

Diversity in what is ultimately funded within and across pillars of health research should also be a measure of quality according to some participants.

Although not specifically related to the peer review system, the following suggestion was offered by one participant. CIHR should consider looking at the relative citation ratio to assess the impact of research and how it can be adapted appropriately for pillar 4 research. For more information, see original paper or the NIH site for doing analyses on papers and researchers.

Date modified:

2017-05-30