Evaluation Plan Guidance Page 60
ADVERTISEMENT
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68EVALUATION PLAN GUIDANCE
SOCIAL INNOVATION FUND
Table 2: Types of Matching for Control and Comparison Group Formation
General Design and Description
Matching Type
Description
Relation to Validity
Single Group
No Comparison
No comparison group is specified.
Single group designs that do not have many data points
In general, single group designs do
Group
across time do not have strong internal validity. Only
not involve a comparison group.
by drawing a large, statistically random sample from
One way of approximating a
the target population can a study with a single group
comparison group is to use an
and no comparison group attain either internal or
interrupted time series design.
external validity.
Interrupted Time
In an interrupted time series design, program participants
In general, single group designs do not deal with
Series
serve as their own comparison group. Relatively large
threats to validity very well. Time series designs do
amounts of data are collected on participants (or groups
deal with some of them; effects related to being in the
or sites) prior to program participation and then again
program and the passage of time are controlled for in
after program participation. This volume of data allows
this design. Biases related to how people are selected to
for the pre-treatment data, or data collected prior to
the program are not well controlled for, and
program participation, to serve as a control for the post-
generalizability (external validity) is weak. Only by
treatment data, the data collected after program
drawing a large, statistically random sample from the
participation.
target population can a single group without
comparison study attain either external validity.
Comparison Group
Matched based on
Comparison group chosen without consideration to
Unmatched comparison groups do not control for
Comparison groups are composed of
Convenience
characteristics of that group in relation to the
selection bias or other issues related to the formation of
individuals (or groups or sites) that
characteristics of the program participant group.
treatment or control groups.
are selected through a non-random
Matched based on
Matching based on a limited number of observed
Matching based on characteristics somewhat controls
means. Different ways of selecting
Characteristics
characteristics of program participants and non-program
for selection bias and other issues related to formation
the comparison group result in
participants, such as age, gender, race, years of education,
of treatment and comparison groups. However,
different levels of statistical
etc. In certain situations, a comparison group initially
because of the limited number of known characteristics
assurance concerning the
formed based on convenience may, after post hoc
upon which the match is based, the comparability
representativeness of the
statistical analysis, be considered a comparison group
between the two groups is less than in other matching
comparison group (which does not
matched on characteristics if the matching is shown to be
methods.
receive treatment or participate in
strong.
the program) in relation to the
Regression
Regression discontinuity implies assigning individuals
Regression discontinuity matching controls for
program participant group.
Discontinuity
into groups by examining a quantifiable indicator related
selection bias and other biases related to group
Comparison group members (or
to the key outcome. If the indicator is reliably and validly
assignment (although not perfectly). Provided both
groups or sites) may be identified
measured, a well-defined cutoff score can be used to form
groups are drawn from the target population and are
through various means.
the program participation (treatment) and comparison
generally representative of it, this form of matching
groups, with those in more need (as measured on the
may lead to generalizable results.
indicator) assigned to the program group and those with
C.11
ADVERTISEMENT
0 votes
Related Articles
Related forms
Related Categories
Parent category: Education