Evaluation Plan Guidance Page 48
ADVERTISEMENT
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68EVALUATION PLAN GUIDANCE
SOCIAL INNOVATION FUND
Discussion of whether the mode of data collection is the same for the intervention and control groups is included.
If administrative records (e.g. school academic/truancy records; unemployment insurance data) will be used, the source and
availability of these data as well as the evaluator’s experience using these data sources are described.
Expected sample sizes at each data collection point are included.
Statistical Analysis of Impacts
If a between-groups design is planned, an Intent-to-Treat (ITT) analysis is described, which compares outcomes between those
assigned program services and those not assigned.
A clear description of the steps of the analysis is provided.
How the statistical analysis of the data is aligned with the research questions is explained.
How the statistical analysis is aligned such that the unit of analysis corresponds to the unit of assignment is described.
The statistical model used to estimate the program effect is fully specified and all variables in the model (and their coefficients)
are defined.
Assumptions of the model are listed.
The program effect model is shown to be consistent with the statistical model used for the statistical power analysis during the
design stage.
Model estimation procedures are included.
If applicable, covariate adjustments to estimated program effects, adjustments for missing data, estimation of standard errors, and
corrections for multiple comparisons are described.
If using a non-randomized between-groups design, statistical methods that adequately control for selection bias on observed
characteristics are described.
Sample Retention and Missing Data
How overall and differential attrition will be calculated and assessed is detailed.
An outline of specific procedures planned for assessing and adjusting for potential biases, due to non-consent and data non-
response, is included.
Any intentions to use multiple imputation for missing data are discussed. This imputation should match the analysis to be
conducted (the imputation model should use multi-level procedures if the analysis is multi-level; if the statistical analysis uses
maximum likelihood or Bayes to incorporate missing data patterns, then this should also be noted).
A brief description of how the plan is designed to minimize missing data with particular focus on minimizing differential attrition
is provided.
Multiple Outcome Measures
If the proposal has multiple related confirmatory research questions, or a single confirmatory question evaluated using multiple
outcomes, adjustments made to reduce the chance of a Type-I error are described.
Human Subjects Protection
If the researchers will work with personally identifiable data, procedures to secure IRB approval are discussed. Plans requiring
IRB approval should include:
IRB(s) that will review the submitted application;
Type of approval sought and process for securing this approval;
Duration that the approval will be in effect, and expected approval date; and,
Timeline for when and how this approval will be secured.
Plans that do not require IRB approval explain why they do not think approval is necessary.
B.8
ADVERTISEMENT
0 votes
Related Articles
Related forms
Related Categories
Parent category: Education