Evaluation Plan Guidance (Page 30 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

The evaluation plan should be clear on the level of statistical analysis (e.g., individual or site) and this analysis

should be at the same level as the random assignment or matching (e.g., individual or site). The analysis needs

to match the level of assignment in order to ensure that evaluators do not make an erroneous inference about

the statistical significance of the program effect (i.e., ascribing a treatment impact on individuals while

analyzing program sites in aggregate). Further, the evaluation plan should include the specification of the

statistical model used to estimate the program effect, including the addition of any covariates, weights, or

other adjustments. Provide the assumptions made in the model and indicate whether any of these assumptions

are likely to be violated. For quasi-experimental and pre-experimental studies, the evaluation plan should also

include a detailed discussion of the statistical methods used to control for

selection bias

on observed

characteristics correlated with the outcomes of interest.

Missing Data

It is to be expected that some participants (or control or comparison group members) may drop out of the

study, the program, or otherwise become impossible to contact. This creates “missing data,” or holes in the

collected data, that need to be dealt with in order for a statistically sound analysis to take place. When

researchers and evaluators refer to missing data, they generally are referring to information that was not

collected, but could have been.

Missing data can be a problem when trying to understand the effects of a program. For example, if only half of

all participants complete an entire program, but all of them show positive change, it remains unclear if the

impact is due to the program or if it is due to the characteristics of the people who completed the program. If

nothing is known about the people who did not complete the program, it would be difficult to say with

certainty that any change found among participants was due to program participation.

Specific Guidance: Missing Data

The evaluation plan should describe how attrition rates will be calculated, both for the study as a whole (in

general, taking into account the number of participants who were continuously part of the study versus the

number of participants who were part of the study only during a particular, truncated period of time). Similar

to the overall attrition rate, differential attrition rates, or the rates at which particular subgroups (e.g., men

versus women) continue to take part in the study or not, should also be addressed in the plan. The plan should

describe how rules will be constructed for deciding how long participants have been in the program in order

for them to count as having completed the program or not, or, if appropriate, what constitutes the various

amounts of program exposure or completion.

To illustrate, it may prove useful to compare participants who completed an entire program with participants

who completed part of the program, as well as the control or comparison group to better understand how

different amounts of exposure or participation affect outcomes. Keeping track of both absolute and differential

attrition can also be helpful for implementation evaluation, too. That is, knowing when participants leave a

program may help identify any strengths or weaknesses of the program.

Sometimes, but not in every case, adjustments are made to data to adjust for

biases

related to missing data.

Describe any procedures planned for adjusting data to assess and/or deal with these biases in the data. In

particular, any use of multiple imputation, the replacement of missing data with substituted values, should be

explained fully and clearly. Data on outcomes of interest should never be imputed, however.

nationalservice.gov/SIF

Evaluation Plan Guidance Page 30

Related Articles

Related forms

Related Categories