Evaluation Plan Guidance (Page 66 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

experimental designs.

Regression: A statistical model used to examine the influence of one or more factors or characteristics on

another factor or characteristic (referred to as variables). This model specifies the impact of a one unit change

in the independent variable or variables (sometimes referred to as the predictor variable or variables) on the

dependent variable (sometimes referred to as the outcome variable). Regression models can take a variety of

forms (ordinary least squares, weighted least squares, logistic, etc.) and require that the data meet certain

requirements (or be adjusted, post hoc, to meet these requirements). Because regression models can include

several predictor variables, they allow researchers to examine the impact of one variable on an outcome while

taking into account other variables’ influence.

Regression Discontinuity Design: A form of research design used in program evaluation to create a stronger

comparison group (i.e. reduce threats to internal validity) in a quasi-experimental design evaluation study.

The intervention and control group are formed using a well-defined cutoff score. The group below the cutoff

score receives the intervention and the group above does not, or vice versa. For example, if students are

selected for a program based on test scores, those students just above the score and those students just below

the score are expected to be very similar except for participation in the program, and can be compared with

each other to determine the program’s impact.

Selection Bias: When study participants are assigned to groups such that the groups differ in either (or both)

observed or unobserved characteristics, resulting in group differences prior to delivery of the intervention. If

not adjusted for during analysis, these differences can bias the estimate of program impacts on the outcome.

Standard Error: In the context of an impact evaluation, this is the standard deviation of the sampling

distribution of the estimate of the program impact. This estimate is divided by the standard error to obtain the

test statistic and associated p-value to determine whether the impact is real or due to chance (i.e., sampling

error).

Statistical Equivalence: In research, this term refers to situations in which two groups appear to differ, but in

truth are not statistically different from one another based on statistical levels of confidence. In a sample, two

groups may have what appears to be an average difference on a baseline characteristic. However, when this

difference is assessed relative to the population from which this group was drawn (using a statistical

hypothesis test), the conclusion is that this difference is “what would be expected”, due to sampling from the

population, and there is really no difference, statistically, between the groups.

Statistical Power: A gauge of the sensitivity of a statistical test. That is, it describes the ability of a statistical

test to detect effects of a specific size, given the particular variances and sample sizes in a study.

Theory of Change: The underlying principles that generate a logic model, a theory of change clearly expresses

the relationship between the population/context the program targets, the strategies used, and the outcomes

(and/or impact) sought.

Treatment-on-Treated (TOT): In contrast to Intent-to-Treat (ITT), Treatment on Treated (TOT) is a type of

D.5

Evaluation Plan Guidance Page 66