Evaluation Plan Guidance (Page 22 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

could lead to an observed intervention effect. Similar to a single case design, interrupted time series designs

are likely to have limited generalizability.

Specific Guidance: Interrupted Time Series Design

If an interrupted time series design is proposed, demonstrate that the number of time points to be measured

prior to and after the intervention is sufficient to establish a trend and rule out rival explanations. Describe the

timing of measures and their appropriateness to the intervention. Plans to include comparison cases should

also be provided, if at all possible. If a comparison case is included, it should be clearly described.

Pre-Experimental Designs

If it is not feasible or appropriate to conduct an experimental or quasi-

Additional Resources

experimental evaluation of a program, an evaluation design targeting

preliminary evidence may be most appropriate. In the vast majority of these

For an

in-depth description of

cases, an initial evaluation of a program’s effectiveness can serve as a

pre-experimental models

see

precursor to a more rigorous evaluation.

hods.net/kb/destypes.php.

Pre-experimental design evaluations are characterized by lack of a control or

statistically matched comparison group, and often incorporate more limited data collection. These include

studies with a single group that uses a post-test, or pre- and post-test and studies with two groups that are not

randomly assigned, statistically matched, or controlled that use post-test or pre- and post-test comparisons.

They also include approaches that address causality through alternative methods, such as intensive case

studies, general elimination methods, and reflexive control designs (e.g., single pre-test with multiple post-

tests).

Specific Guidance: Pre-Experimental Designs

If an evaluation using a pre-experimental design and targeting preliminary evidence is proposed, the

evaluation plan should:



Provide justification for its use and detail reasons why an impact evaluation with fewer threats to

internal and external validity cannot be conducted;



Describe the full study design in detail along with any assignment to groups, and outline the potential

of non-equivalence of groups;



Explain the treatment and any counterfactual conditions, as needed;



Provide details of any additional threats to internal and external validity of the design; and



Indicate plans, if any, to conduct an impact evaluation meeting criteria presented above, with attention

paid to the ways in which the proposed research study will inform and facilitate this future research.

Feasibility Studies

In some evaluation plans, in conjunction with an implementation evaluation and/or with an outcome or

impact evaluation, a program may conduct a feasibility study in preparation for conducting a more rigorous

evaluation. Such a study might be appropriate if a program is developing new delivery mechanisms, creating

additional content, or expanding services to a new organization or institution. A feasibility study is a

preliminary study designed to test measures, instruments, data collection techniques, or other aspects of an

outcome or impact evaluation.

nationalservice.gov/SIF

Evaluation Plan Guidance Page 22

Related Articles

Related forms

Related Categories