Evaluation Plan Guidance (Page 62 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

Appendix D: Glossary

Alpha Level (α): The criterion that allows researchers to use the p-value (see below) to determine whether an

estimated program impact is statistically significant. The p-value is the probability that an estimated impact

that large, or larger, in magnitude could have occurred by chance, even if the program had no actual

impact. The alpha level should be specified by the researcher before outcome data collection begins. Many

researchers set alpha equal to 0.05, by convention, but under certain circumstances a larger value (such as an

alpha level of 0.10) or a smaller value (or an alpha level of 0.01) may be preferable.

Baseline Differences: For between-group designs, the difference between the average characteristic of one

group versus the average characteristic of another, prior to program (or intervention) delivery. A statistical

hypothesis test is typically applied to evaluate whether this difference is due to chance.

Between-Group Designs: Designs that compare the average outcome from each of at least two groups.

Bias: In the context of program evaluation, this refers to the extent to which the program impact estimated

using the study sample approximates the true impact in the population, across many replications. When an

estimate is biased, it will be higher or lower than the true impact.

Blocking: This approach is used during the assignment phase of a study to improve precision of the estimated

program impact, to balance the groups on certain characteristics, or both. This is accomplished by determining

a characteristic (such as locale), then ordering study units by levels of that characteristic (e.g., urban, suburban,

and rural). Within each level, study units are assigned to groups (using random assignment or matching).

Comparison Group: A group of study participants who do not receive program services, usually formed

through methods other than random assignment. This group serves as the counterfactual relative to the

program (or intervention) group. Because this group is formed by methods other than random assignment, it

is considered a “weaker” comparative group than a control group formed by random assignment.

Confirmatory Research Question: The primary research question that the study is statistically powered to

address and the answer to which can be used to inform policy.

Control Group: A group of study participants, formed by random assignment, who do not receive program

services, and is assessed in comparison to the group receiving services (or the intervention). A randomly

assigned control group of participants, statistically, should be similar in both known and unknown ways to the

group of participants receiving services. It is considered the strongest possible group to compare to the

intervention group.

Correlation: A statistical relationship between two characteristics that vary in a continuous way, such as a

relationship between number of hours receiving training and wages after completing a program. This term is

used to describe how one characteristic of study participants or group of study participants varies with

another characteristic of the same study participants (or group of participants). This covariation is typically

D.1

Evaluation Plan Guidance Page 62