Evaluation Plan Guidance (Page 23 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

Specific Guidance: Feasibility Studies

Describe the barriers to implementing a full-scale study yielding moderate or strong evidence and provide a

complete description of the feasibility study to be conducted, including, as appropriate, study elements related

to a description of the program, engagement with stakeholders, or assessments of data availability and

quality. The evaluation plan should address the plans to test the practicality of the use of a control or

comparison group or other counterfactual. Describe how the feasibility study prepares the program and the

evaluators for planning, implementing, and completing a more rigorous evaluation.

Combination of Designs and/or Analyses

In some situations, using a combination of several study designs may be useful to capture the evidence of

effectiveness of the program. For example, a small scale RCT may be used alongside a larger matched

comparison study with groups formed by propensity score matching.

Specific Guidance: Combination of Designs and/or Analyses

If multiple study designs or analyses are combined, the proposal should provide clear details of all

components, in accordance with the specifications provided above. Further, the proposal should provide a

rationale for using a combination approach, with a focus on how using multiple analyses offsets threats to

internal and external validity.

Note that the design types discussed above are not intended to be a full list, and other design types and

combinations of design types appropriate to your program and evaluation goals are welcome.

Implementation Evaluation

Implementation evaluation is an assessment of how well a program does what

Additional Resources

it sets out to do. Rather than focusing on the outcomes, however,

implementation evaluations focus on the process by which a program provides

See Weiner (2004) for

services or otherwise accomplishes its mission. Implementation studies center

information on

implementation evaluation

on discerning how closely the actual running of the program matches the

research design.

theory that generated both the program in general, as well as the particular

components that participants experience.

Specific Guidance: Implementation Evaluation

Implementation studies are strongly encouraged as part of any evaluation plan. The program theory and logic

model should guide the implementation evaluation. An implementation evaluation examines both how well

the program matches the theory behind its creation, and what the program actually does on the ground. In

addition, an implementation evaluation strives to uncover the exact processes that lead to outcomes. The plan

should measure variation in implementation within the program as well as across program sites, as warranted.

The plan should assess both the quantity and quality of services delivered to the target population.

Additionally, the plan should examine if the program is indeed serving the people for whom it was intended,

and if appropriate, if there is a control or comparison group who may or may not be receiving similar services

through another program or programs.

nationalservice.gov/SIF

Evaluation Plan Guidance Page 23