Evaluation Plan Guidance (Page 28 of 68) in pdf

EVALUATION PLAN GUIDANCE

SOCIAL INNOVATION FUND

participants complete surveys, will project staff maintain records that will be used, or will administrative data

such as academic transcripts be used? It is likely that some combination of efforts will be required to capture

all the data the measures warrant.

The way in which data are collected may not be identical for both program participants and control or

comparison group members, but ideally the same data will be collected across both groups. Indicate any

differences in data collection between the two groups, including sources, means of data collection, and who

will collect the data.

Finally, include a timeline outlining the data to be collected at various points (for example, what data are

collected prior to program participation, during, and after). Indicate expected sample sizes at various points in

time in the data collection process.

Statistical Analysis of Impacts

To ensure the strongest possible evidence results from the evaluation, the correct statistical analysis techniques

must be employed. The statistical technique chosen will depend on the types of research questions and

outcomes or impacts specified in the research design; they will also depend on the type(s) and quantity of data

collected. For example:



Studies that collect data across time will need to use statistical models that are appropriate for such

data (e.g., fixed effects models);



Research designs with comparison groups may need to statistically control for attributes of group

members (e.g., regression models); and



Designs that involve nested data, such as students within classrooms will need models that handle that

type of interaction (e.g., hierarchical linear models).

Different types of data will also necessitate different statistical models. For example, determining if someone

uses more or less service after program participation implies that the outcome

measure varies along a continuous spectrum (i.e., number of visits or service

Additional Resources

utilizations). This type of question would require the use of a model that fits

See Schochet and Chiang

this type of data and that can also adequately take into account other factors

(2009) for information on

besides program participation (ideally) to ascertain the impact of program

causal inference and

participation on the number of visits (e.g., a linear

regression

or related

instrumental variables

model). Conversely, if the outcome of interest is whether or not a student

framework in TOT

enrolled in college, the outcome has only two categories (enrolled or did not

randomized clinical trials.

enroll), and the statistical model must be appropriate for that type of

categorical data (e.g., a logistic regression or related model).

More details on specific, commonly used statistical models are detailed in Table 1 in Appendix C (Examples

and Templates) of this document.

Intent to Treat (ITT) and Treatment on Treated (TOT) Analysis Frameworks

Two main perspectives exist for guiding analysis in evaluations. If the proposed study has a randomized

between-groups design, an

intent-to-treat

(ITT) framework is suggested (though not required). This

framework starts with the premise that analytically, evaluators are assessing outcomes based on program

components rather than participant experience. This framework is useful because it (1) requires the evaluator

nationalservice.gov/SIF

Evaluation Plan Guidance Page 28

Related Articles

Related forms

Related Categories