Evaluation & Performance Measurement

A formative evaluation is done while a program is running, to improve its design and delivery. A summative evaluation is done once the program has operated long enough to judge its outcomes, and informs decisions to continue, change or end it. A process evaluation looks specifically at whether the program is being delivered as planned. Qatalyst conducts all three types and helps clients choose the one that fits the decision they face.

A performance measurement framework is a plan for tracking whether a program is producing its intended results. It usually includes a logic model, indicators for outputs and outcomes, targets, data sources, collection frequency and who is responsible. Qatalyst develops these frameworks for programs and portfolios. After evaluating the First Nations Health Authority, it helped the organization build a performance measurement framework and evaluation strategy for its new strategic plan. Read the FNHA case study.

The length of a program evaluation depends on the number of lines of evidence, the quality of existing program data, how many people must be surveyed or interviewed, and the client's review and approval steps. Qatalyst runs evaluations in three phases: planning with the client's steering committee, a research program, then analysis and reporting. Its published evaluation case studies were each completed on time and within budget.

Most program evaluations combine several lines of evidence, typically a document and data review, a literature review, surveys, key informant interviews, focus groups and case studies, so that findings can be checked against each other. For the Government of Yukon's Respectful Workplace Policy, Qatalyst surveyed 262 clients, held follow-up interviews with 67 of them, interviewed 34 key collaborators and ran two focus groups, leading to seven recommendations. Read the case study.

An evaluability assessment checks whether a program is ready to be evaluated in a useful way. It reviews whether the program's objectives and logic model are clear, whether the needed data exists or can be collected, and what questions decision-makers need answered. The result is advice on whether to evaluate now, what design to use and what to fix first. Qatalyst offers evaluability assessments alongside evaluation frameworks and evaluations of all sizes.

A Credentialed Evaluator (CE) is an evaluator who holds the professional designation granted by the Canadian Evaluation Society. Applicants must show relevant graduate-level education, evaluation work experience and competence across the society's evaluation competencies, and must keep up professional development to maintain the designation. Members of Qatalyst's evaluation group are Credentialed Evaluators.