How do you evaluate the efficacy of an organization active in nine domains of public health with programs worldwide, public and private sector partnerships, and more than 100 staff?
This was the question posed in 2016 by APHL’s Institutional Research Program as it embarked upon a six-month process of research, discussion and analysis to develop a comprehensive evaluation plan for the association. APHL would use this plan to assess the impact of its work and to inform decisions about future programming.
Planning for the Evaluation
Essential to plan development were the knowledge and experience of APHL members, funders and staff, but some were unfamiliar with evaluation approaches. As a result, training in the basics of evaluation became the first step in plan development. Others included:
- Key informant interviews to understand APHL’s need for evaluation
- Focus group discussions to assess the funder’s needs for evaluation findings
- Online surveys to solicit evaluation questions
- Document reviews of reports and literature
Then came the challenge of selecting an evaluation approach. Traditional “experimental” methods can assess the impact of a product or intervention, but not a complex organization. What’s more, APHL’s evaluation method had to identify how and why change had occurred in addition to whether it had been effected. After much study and discussion, APHL opted for a theory-based evaluation approach, which lays out how an organization plans to bring about needed changes (outcomes) and then tests that theory.
Utilizing a Theory of Change Model
The APHL theory of change postulates that better population health outcomes will result from evidence-based decision making as a consequence of improved surveillance. Strong surveillance is in turn predicated on the existance of a functioning public health laboratory system that:
- Is able to provide timely and reliable data
- Works effectively with other public health systems.
And what, exactly, is required to achieve a functioning public health laboratory system? Per APHL’s theory of change, the answers are:
- Public health laboratory development in the areas of pre-analytics, improved methods, organizational sustainability, biosafety and biosecurity, and workforce
- Better defined laboratory networks to handle a wider range of current and emerging public health issues
- Accurate, efficient and timely data exchange within and outside the laboratory network.
APHL pursues its goal of a vital public health laboratory system by applying the strategies outlined in its theory of change. They include:
- Research and analysis to understand the capabilities, trends, needs and contributions of public health laboratories
- Convening and collaborating with member public health laboratories and their partners
- Being the voice of public health laboratories as a bridge to key stakeholders
- Technical assistance and service to improve the capacities and capabilities of public health laboratories
- Innovating and championing ideas that will move the public health laboratory system forward
- Educating and informing to enhance the competencies of public health laboratories and share best practices
- Facilitating private/public partnerships to increase technology transfer and improve resource utilization
Transforming Theory into Practice
The theory of change model is evolving to become an integral part of APHL’s organizational culture. The association is beginning to apply the model to strategic and programmatic planning, while individual programs are planning to use it to track progress and share achievements. Yet the theory of change model is far from final. It will be refined and updated continuously with feedback from stakeholders and findings from APHL evaluation activities.