Tackling the challenge: evaluating the funder’s role in KT

2 October 2019

Research funders and other research organizations are increasingly challenged to be more evidence-based and strategic about their knowledge translation (KT) activities and their potential for return on investment. MSFHR is no exception – as a data-driven organization, we want to know that our approach to KT is having the intended impact.

In this blog post, Gayle Scarrow, MSFHR’s director of knowledge translation, shares some of the lessons learned after a recent evaluation of our KT activities and how those learnings will inform our work going forward.

Forward Thinking is MSFHR’s blog, focusing on what it takes to be a responsive and responsible research funder.

Tackling the challenge: evaluating the funder’s role in KT

In an earlier Forward Thinking blog, our President & CEO, Dr. Bev Holmes, wrote about the need for funders and other research organizations to be more evidence-based and strategic about their knowledge translation (KT) activities and their potential for return on investment. As Bev wrote, “Ultimately, we need to understand where we can have the most influence, how much should we do in which areas, and for what benefits and impacts.”

My team and I really wanted answers to these questions – like all funders, MSFHR is being challenged to increase the impact of the health research we fund and our KT work is a big part of how we’re doing this. Last year, we undertook an evaluation of our KT theory of change by looking at our KT activities over the past three years (2015-2018).

Now that work is complete, let’s see if we can answer Bev’s questions.

Evaluation & our theory of change for KT

Our model of the theory of change (ToC) for KT is made up of five areas we believe create an environment for effective KT:

Building capacity for KT (e.g. training and resources for both researchers and research users)
Doing KT/managing KT projects (e.g. researcher-decision maker forums, other events)
Funding KT (e.g. awards)
Advancing the science of KT (e.g. involvement in research)
Advocacy for KT (e.g. encouraging and influencing system change)

Additional areas that support the model are: assessment of KT needs to identify gaps and opportunities, evaluation of KT activities (ours and our funded researchers) and communication of KT activities (again both ours and our funded researchers).

Based on our evaluation framework, we focused on short-term outcomes of our KT capacity-building activities¹ and to a lesser extent, KT-focused awards² for the period of 2015-2018. While we work in all areas of our ToC for KT model, these two areas reflect the bulk of our KT activities.

We learned very quickly that data collection was going to be challenging. We planned to use a survey, key informant interviews and existing monitoring data on program reach, usefulness and use (e.g., training and webinar participant feedback) to inform our work. We expected data in some areas would be less robust then in others and we were right – existing monitoring data was of limited use. We also had a low response rate for the survey, uneven geographic distribution of survey respondents, and survey respondents who were not able to confidently answer outcome-related questions about newer aspects of our KT program (particularly awards). Unfortunately, no other quantitative data was available to address outcome-related questions and we had to be careful when interpreting the quantitative results that did exist. Key informant interviews, however, provided rich qualitative data that helped us understand the effectiveness, value, and interconnectedness of our KT program activities in meeting overarching goals.

Based on these learnings, and to inform future evaluations, we’re developing consistent indicators and data collection mechanisms for our KT capacity-building activities that are realistic in terms of our available resources and that won’t overly burden our stakeholders. For example, we will focus our evaluation efforts on KT Pathways and our upcoming Train-the-Trainer program, activities that require significant time and resources to develop and maintain, and that have significant potential for outcomes and impacts. To a lesser degree, we will ensure we collect more consistent evaluation data on our other KT activities (e.g. Health xChange; KT Connects webinars; KT Collaborative) to ensure they are effective in meeting their intended goals.

What did we learn?

The good news is our KT program is achieving its goals:

Raising awareness, knowledge and skills in KT
Increasing collaborations between researchers and research users
Improving the application of KT principles by researchers and research users
Promoting/advocating for KT
Increasing the pool of KT experts in BC
Generating and disseminating new KT knowledge

Stakeholders are very positive about our program’s contributions to the health research ecosystem in BC and value the award and capacity-building components of the program. Interestingly, the evaluation also found our KT program is having success in other, unanticipated ways including:

PositioningMSFHR as a provincial and national leader in KT
Raising the profile of KT in BC
Catalyzing a KT community of practice in BC
Empowering KT practitioners (e.g. knowledge brokers, KT specialists, KT practice leaders) to do their jobs well, and
Supporting the creation of a common language and understanding of KT in the funding context

This suggests that our efforts are driving a culture change in BC towards an increased use of research evidence through KT. For MSFHR, that directly relates to our organizational commitment to maximize the value and impact of our funding through evidence-informed health care and health system change.

The key informant interviews identified six key enablers of our overall KT program impact that nicely reflected several components of our theory of change for KT. Listed in order of thematic prominence, the six key enablers are:

Funding KT-specific awards
Our leadership activity in messaging the importance of KT
Our work as a provincial coordinating body for KT
The suite of flexible, high-quality KT capacity-building resources offered by MSFHR
Frequent communications that keep KT ‘on the radar’ in BC
The perceived seamlessness and coherence of our KT program with no obvious gaps or inconsistencies

For those who want to dive into the evaluation findings in more detail, you can find the report summary here.

The effectiveness of our theory of change

The data we’ve collected to date indicates our theory of change for KT is effective; the key informant interviews confirm that all five areas of our KT model are intertwined. In relation to our logic model, at least three areas are both outcomes and activities depending on context. For example, building KT capacity is both an outcome of longer-term advocating for KT and an activity (providing KT training and resources). This means the model can continue to guide development of our KT strategy and activities.

The relationship between the functional areas of our KT model, as indicated from this evaluation, is shown below. The most critical elements of the model are shown in bold, and the most important relationships are shown in large arrows.

Where could we improve?

Our KT evaluation work was a valuable opportunity to learn where we could improve. While our key informants generally felt our KT priorities were correct and that no significant changes are needed; they did highlight some areas where we could do better:

Delivering more capacity-building activities (particularly webinars and workshops) for audiences outside Vancouver (we were perceived by some respondents as being primarily Vancouver-centric).
Improving our promotion of the range of KT services and supports we offer, broadening the scope to include a wider-range of KT practice areas, and targeting a wider range of audience experience levels.
Broadening the target audiences for KT awards and raising awareness of them.

We’ve started to address some of these areas and will be working on others in the coming months. For example, working with BC-based partner organizations we co-developed and implemented KT Pathways, a pan-provincial resource for KT learning and professional development for researchers, research users and knowledge brokers.

With our partner Arthritis Research Canada, we’ll be reaching out to stakeholders in BC for their advice on what the third year of KT Connects – our partnered webinar KT series – should look like (spoiler alert – it will feature more non-Vancouver experts), and we’re developing a KT communications plan to raise awareness of the range of KT supports available on our website.

Did the evidence answer our questions?

Based on the evaluation evidence, can we answer Bev’s questions? Do we understand where we can have the most influence, how much we should do in which areas, and for what benefits and impacts? Yes and no!

We’ve heard from key informants that our suite of KT resources, rather than any one activity, has enabled the success of our program. We have evidence that indicates our theory of change for KT is effective and that the three most critical areas of our KT model are building KT capacity, funding KT, and advocating for KT. We also heard that all five areas of the model are intertwined so it’s important we continue work in all of the areas including advancing KT science and managing KT projects.

We don’t have consistent project level data that can inform an extended activity analysis so we’re working on that for future evaluations. Fortunately, the KT program evaluation findings are not the sole source of evidence informing our work – we’re also collecting data, for example, on how our peer review panels assess the KT elements of funding applications to better understand what supports they may need. This work builds on our observation of the need for reviewer training to assess KT.

Looking at the current number of KT workshop opportunities in the province, we have realized there are not enough KT trainers in BC to meet the anticipated demand for training arising from the launch of KT Pathways. To address this, we’re developing a KT train-the-trainer program to increase the number of individuals in BC able to offer KT training workshops. We anticipate the program will result in broader impacts then the one-off workshops we’ve been able to offer in the past and that pan-provincial initiatives such as KT Pathways and train-the-trainer will provide a better return on investment of our resources.

Where to next?

We’re taking what we learned from the evaluation of our KT program as one of the first inputs to a new five-year, evidence-based organizational KT plan. We’re delving into the recent literature on KT as it relates to the funders’ role, and will be consulting with stakeholders about the complex and changing health research ecosystem in BC. We’ll be looking for opportunities where we can partner, leverage and facilitate change towards the increased use of health research evidence in practice, policy and further research.

An invitation

Funders and other research organizations need to use evidence-based KT strategies and activities so that they can understand the potential for impact, effectiveness, and efficiency of their efforts.

We’d love to work with other funders and research organizations locally, nationally and internationally to build the evidence-base needed to inform strategic KT program planning. Please send me an email or give me a call if you’re interested in working with us.