Evaluation Plan Guidance (Page 65 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

internal and external validity, with evaluations that do a better job generating strong or moderate levels and

those that are less able to do so generating preliminary levels of evidence.

Matching: A technique used to pair participants in an evaluation based on observed characteristics that are

correlated with the outcome of interest. The pairing is then used to create intervention and control groups that

are similar based on these characteristics.

Minimum Detectable Effect Size (MDES): The smallest effect size (usually, the comparative difference

measured in an outcome between program participants and control or comparison group members) that can

be detected for a given design and under certain assumptions with a specified probability (typically .80).

Typically, increasing the sample size leads to a smaller MDES (that is, enables the study to detect a smaller

impact).

Multiple Comparisons: When between-group designs are used, there are opportunities to compare multiple

groups on the same outcome or two groups (program participants versus control or comparison group

members) on multiple outcomes. This comparison can artificially inflate the alpha level and require the

researcher to adjust it downward. That is, if many outcomes are addressed in a study, it is possible that some

will be erroneously viewed as statistically significant even though they are in reality due to chance.

Nomothetic: In contrast to idiographic, nomothetic focuses on group outcomes typically based on the average.

Post Hoc: Means “after the fact.” In the context of evaluation, the term refers to analysis of data that was not

specified prior to analyzing the data.

Propensity Score: A score calculated using logistic regression techniques based on known characteristics of an

individual or group, which predicts probability of group membership (e.g., intervention or program

participation group versus comparison group).

Propensity Score Matching: The use of propensity scores to identify participants for inclusion in the

comparison group. Propensity score matching can decrease pre-treatment differences in the treatment and

comparison group, thereby reducing selection bias, which constitutes a key threat to internal study validity.

P-values: In the context of an impact evaluation, a statistical term used to describe the probability that the

impact observed in the sample could have come from a population in which there is no impact.

Quasi-Experimental Design: A design that includes a comparison group formed using a method other than

random assignment, or a design that controls for threats to validity using other counterfactual situations, such

as groups which serve as their own control group based on trends created by multiple pre/post measures.

Quasi-experimental design, therefore, controls for fewer threats to validity than an experimental design.

Random Assignment: A process that uses randomly generated numbers (or other method of randomizing

study participants) to assign study units (people, program sites, etc.) to either the program participant or

control group. The use, or lack of use, of this process differentiates experimental designs from non-

D.4

Evaluation Plan Guidance Page 65