Evaluation Plan Guidance (Page 19 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

For a randomized design, provide comprehensive and clear information as to what constitutes taking part in

the program (i.e., what constitutes treatment or exposure, does some program participation count), and what

the control conditions will be (i.e., no knowledge of the program, not participating in some parts of a

program). Fully describe the randomization process, including how individuals will be assigned to program

participation, how the random numbers will be generated, who will conduct the random assignment, and any

matching that might be used prior to randomization. The unit of randomization should be the same as the unit

at which the outcome is measured; this means that if individuals are the

unit of

analysis, they should be

randomly assigned to each group, but if program sites are the unit, the sites should be randomly assigned to

each group.

Non-Randomized Group Designs – Groups Formed by Matching

Sometimes it is not feasible to randomly assign potential program participants (or groups or sites) to treatment

and control groups. In these situations, a comparison group can be formed by matching study participants, or

clusters of study participants, on a set of pre-intervention measures of the program outcome (e.g., pre-test

scores for an academic program, pre-participation employment status for a job related program) and/or pre-

intervention measures that are likely correlated with the program outcome, and/or other characteristics. The

main goal is to have two groups of individuals (or sites) that are as similar as possible on as many

characteristics as possible.

These types of designs are called

quasi-experimental

because they limit the

evaluator’s ability to make causal claims about a program’s impact, compared

Additional Resources

experimental designs

with random assignment. This is because the groups

See Rossi, Lipsey, and

formed by non-random methods will be, at best, equated on measured

Freeman (2004) for a general

characteristics only, whereas random assignment ideally randomly distributes

overview of research design in

unmeasured characteristics, such as motivation, between the treatment and

evaluation.

control groups. Working with a

quasi-experimental

design, the evaluator can

attribute the observed effect on the outcome to the program, but with

See Shadish, Cook, and

reservations, because unmeasured characteristics may unknowingly be

Campbell (2002) for details on

responsible for the outcomes observed after program participation (Shadish,

experimental research design.

Cook, & Campbell, 2002; Rossi, Lipsey, & Freeman, 2004). However, matching

in advance of the treatment with variables that are either pre-intervention

For more information on

quasi-experimental evaluation

measures of an outcome or are highly correlated with the outcome will

designs

see:

minimize the chance of differences between the treatment and comparison

group. Doing so lessens the possibility that observed differences are in fact

/Chp_4.pdf

due to extraneous differences between the treatment and comparison groups,

rather than due to program participation.

Methods for matching people or sites to groups differ in their effectiveness.

Propensity scoring

methods, or

statistical assessments of similarity among participants, are preferred when groups are matched with multiple

pre-intervention measures. However, the type of matching algorithm used to implement the propensity

scoring should be carefully selected based on simulation studies, previous research that demonstrates the

validity of the algorithms, and the goals of the evaluation. See Song and Herman (2010) for information and

guidance on methods of matching for quasi-experimental designs.

nationalservice.gov/SIF

Evaluation Plan Guidance Page 19

Related Articles

Related forms

Related Categories