Evaluation Plan Guidance (Page 21 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

designs can do a good job of evaluating how a program affects participants

(internal

validity) approaching that

of experimental designs, but use of a single case limits the generalizability

(external

validity) of the results to

other groups. One form of the single case design—repeated measures taken prior to treatment delivery and

repeated measures after treatment delivery—is a form of the interrupted time series, but the time series is

typically measured on one individual.

Specific Guidance: Single-Group Designs - Single Subject (or Case)

If a single group design is proposed, each intervention stage of the design should be detailed, including the

baseline stage. The number of measurement points at each phase should be adequate such that trends can be

established and threats to internal validity can be minimized.

Interrupted Time Series Design

Occasionally, an evaluation examines a single group of individuals before and

Additional Resources

after participation in an intervention, with no attempt at controlling who is

The Department of

part of the group. This form of evaluation, referred to as an interrupted time

Education’s What Works

series design with a single group (such as a school or classroom), attempts to

Clearinghouse

(at

capture any change that occurred to the individuals after program

)

participation by examining the general trend found in multiple measures of an

provides many resources

outcome over time.

related to evaluation design.

Data collected across time are referred to as longitudinal data (in comparison

description of interrupted

to cross-sectional data, which are collected at one point in time). Longitudinal

time series designs

is given by

data are defined as repeated measurements on individuals (sites, groups, etc.)

Imbens and Lemieux (2008),

over time. Many researchers use the specific term “time series” to refer only to

(available here:

data collected over time from different sets of individuals (as compared to

t/classes/eco7377/papers/imbe

“longitudinal data,” which refers to data collected from the same individuals

ns%20lemieux.pdf)

over time). However, the term “interrupted time series” is used commonly in

evaluation research to refer to a specific form of longitudinal data analysis,

See Bickman and Rog (2009)

and so is used here to refer to all types of data used in this type of evaluation.

for information on many

The frequency of these measurements and units, and the length of time,

different evaluation designs.

depends on the program or policy being assessed. Simply put, repeated

measurements are made on units prior to implementation (the comparison

See Khandker, Koolwal, and

group) and after implementation (the treatment group). The interruption in

Samad (2010) for information

the time series (or patterns before and after program implementation) can be

on propensity score matching

and regression discontinuity

estimated graphically or statistically (Imbens & Lemieux, 2008).

design.

The interrupted time series design has been most frequently and effectively

See Shadish, Cook, and

used when the intervention is a program or policy of significant scope, such as

Campbell (2002) for citations

community interventions to improve child-rearing practices or mandatory

of studies that have used the

seat belt laws in a state. The social evaluation research literature includes

interrupted time series design.

many descriptions and examples of the development and implementation of

this design and the analysis of data generated by it (Bickman & Rog, 2009;

Khandker, Koolwal, & Samad, 2010; Rossi, Lipsey, & Freeman, 2004; Shadish, Cook, & Campbell, 2002).

Additionally, although this design is listed as a single-group design, a comparison group is usually needed to

address the possibility that extraneous events linked to how data were collected or outcomes were developed

nationalservice.gov/SIF

Evaluation Plan Guidance Page 21

Related Articles

Related forms

Related Categories