Evaluation Plan Guidance (Page 29 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

to compare the outcomes of participants (or groups of participants) between

Additional Resources

those who were assigned to receive program services and those who were not

assigned to the program, and (2) generates an unbiased estimate of the

More information on and

average program effect for participants offered the program. The ITT

examples of statistical

framework is often used because the impact estimate is the one that is often of

analysis can be found on the

UCLA statistical computing

most interest to policymakers (Bloom, 2005).

website:

However, in practice, interventions are more likely to be implemented with

xamples/default.htm.

varying degrees of fidelity to the intended implementation plan. Similarly,

levels of participation can range from participants not taking part at all to

those completing all aspects of a program. The unbiased estimate from an ITT analysis framework will reflect

this range. For this reason, the impact estimate from the ITT is often considered the most policy relevant

because it is based on the experimental structure of the data (i.e., results depend upon either participating, at

any level, or not at all).

The

treatment-on-treated

(TOT) analysis, which is based on what participants actually experience, must

typically be obtained from an extension of the non-experimental structure of the data (Bloom, 2005). Moreover,

the unbiased estimate of a program’s effect on an outcome can be compromised by non-random missing data

on the outcome. This possibility, and how it will be handled if it occurs, should be addressed in the evaluation

plan under missing data.

Conversely, TOT analysis typically requires the evaluator to analyze data collected on individuals based on

their level of program participation. Because program participants self-select their level of program

participation, analyses that estimate program impacts according to these levels are by nature quasi-

experimental.

Specific Guidance: Data Analysis

Open the statistical analysis section of the evaluation plan by describing the ITT analysis framework and then

describing if this framework will be supplemented with another framework, such as TOT. For designs that

form groups through other approaches, such as matching, the principle of beginning with an ITT framework

still applies and should be reflected in the evaluation plan. For example, when matching is used to form

groups prior to program delivery, individuals (or groups of individuals) should be analyzed in the groups

they were in at the time of matching.

Provide clear details of the data analysis used to determine program effects. Describe the types of statistical

analysis to be undertaken, specifying descriptive analysis and/or inferential analysis as appropriate. Include

descriptions of statistical models. Note the

covariates

– the characteristics or variables that will be used in the

model – in the discussion, and clearly delineate the sample (full or partial; including anticipated size) to be

used for each analysis.

List and describe the statistical procedures to be used in analysis (for example, describe the OLS regression,

logistic regression, ANOVA, or whatever model fits the analysis plan best). Table 1 (in the Examples and

Templates section [Appendix C] at the end of this document) lists common statistical models used in

evaluation design along with examples of their application. The reasons why procedures were selected,

including the study design and outcome(s) considered, should also be described. Importantly, the statistical

procedures proposed must align with the research questions, and correspond to the power analysis used to

determine the minimum sample size.

nationalservice.gov/SIF

Evaluation Plan Guidance Page 29

Related Articles

Related forms

Related Categories