Simple randomized evaluations

What is the appropriate level or unit of randomization?
What is the appropriate method of randomization?
How would we implement the randomization?

Study sample & control

Ensure ethical sound selections.
Study sample: Students – men and women, different ages.
Sample size: All students: 200 per year.
External validity:
i) Ensure that a person asked is representative of the group and that we would expect the same answer if we asked someone else.
ii) Reflect over if the results of our randomized evaluations is generalizable to other contexts.

Control: Who, where, how many?

Reflections on ethically sound selections

Without denying access: It can be possible to conduct a randomized evaluation without denying access to the intervention. For example, we could randomly select people to receive encouragement to enrol without denying any interested participants access to the intervention.

Two versions of an intervention

It may be useful to compare two different versions of an intervention, such as an existing version and a version with a new component added.

Quantitative analysis – Statistical method

Basic statistics: The basic assumption to be made is that a set of data, obtained under the same conditions, has a normal or Gaussian distribution. The primary parameters used are the mean (or average) and the standard deviation, and the main tools F-test for precision, t-Tests for bias, Linear correlation and regression and Analysis of variance (ANOVA).
Simple comparison: With randomized evaluations, the simplest method is to measure the average outcome of the targeted group and compare it to the average outcome of the control group. The difference represents the programme’s impact. To determine whether this impact is statistically significant, one can test the equality of means, using a simple t-test. One of the many benefits of randomized evaluations is that the impact can be measured without advanced statistical techniques.
Propagation of errors: The final result of a Programme is calculated from several activities (outputs) performed during the implementation and the total error in a programme is an adding-up of the sub-errors made in the various steps. The bias and precision of the whole Programme are usually relevant parameters.

Qualitative assessment – probability methods

With qualitative assessments, and contrary to statistical methods, the quality of the evidence is not judged by the sample size (the number of observations) but rather the probability of observing certain pieces of evidence. Qualitative impact evaluation includes assessing the contribution made by a particular intervention in achieving one or more outcomes, commonly referred to as a ‘contribution claim’. TestE benefits from process tracing to assess our Strategy for Change and from Contribution tracing to examine the contribution by external stakeholders . We also address Team operations, Cost-benefit, Needs driven, Equal partnership and Unexpected effects.