Evaluation Plan Guidance (Page 16 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

Impact Evaluation

Impact evaluation

designs address the issue of how an intervention is related to changes in its participants or

beneficiaries, and ideally provide evidence about whether it causes the observed changes. The strength of

evidence of an impact evaluation is determined by the extent to which it maximizes

internal

and

external

validity.

Internal validity refers to the ability of the evaluation findings to accurately reflect the impact of the

intervention on participants or beneficiaries. Research designs are thought to have good internal validity if

they incorporate design features that limit the extent to which the changes seen in the study could be caused

by things other than the intervention being measured. These alternative causes could include:



Who gets into the program (or

control

comparison

group) and who stays in each group during the

study;



Things that happen during the time window of program participation that are not part of the program

components (e.g., children get older, the economy rebounds);



The evaluation process (e.g., completing measurement instruments, being observed); and



Decisions made by evaluators and the impact of their actions (e.g., unreliable instruments, instruments

that are invalid for the intended purpose).

Strong external validity occurs when a study’s findings can be generalized to a diverse target population. Even

if the population in question is a targeted one, external validity pertains to diversity in time (i.e., the program

is effective across several years), and location. Designs that have strong external validity adequately address

concerns about the following:



How well the findings apply to other settings, sites, or institutions; and



How well the findings apply to other groups of people, who may or may not be like those in the

current study.

Evaluation designs that minimize threats to internal validity and maximize external validity are most likely to

yield strong evidence. Designs that address key threats to internal validity, although they may not have strong

external validity, can yield moderate evidence. Designs that do not sufficiently address threats to internal

validity will yield preliminary evidence, as will designs that do not incorporate a

counterfactual

scenario (i.e.,

a comparison with what would happen in the absence of the program). For more details, see the information

on “Internal and External Validity and Strength of Evidence” below.

Internal and External Validity and Strength of Evidence

Additional Resources

Strong Evidence: Different types of evaluation designs are likely to yield

different levels of evidence of program impact. These designs are likely to

For more information on

maximize both

internal

and

external validity

if they use reliable measures,

design types and threats to

take into account pre-existing participant characteristics, have program

validity see Table 2 in

participants (or groups or sites) receive identical services, and include a large

Appendix C (Examples and

and diverse sample drawn from the target population in both the group that

Templates).

participates as well as the group that does not.

nationalservice.gov/SIF

Evaluation Plan Guidance Page 16

Related Articles

Related forms

Related Categories