Abstract
This is an impact assessment report of an independent evaluation of the Aga Khan Foundation's (AKF) East African Quality in Early Learning (EAQEL) initiative to determine whether the initiative improves learning outcomes in the early grades (1-3) in two districts in Kenya and two districts in Uganda as was intended. The initiative is also referred to as the Reading to Learn (RtL) approach. The districts covered by the study are Kwale and Kinango in Kenya and Amolatar and Dokolo in Uganda. The four districts were selected by AKF because they were consistently performing poorly in the national examinations in both countries. The EAQEL initiative has two components: Core model and Core model plus. The Core model involved a 'new' instructional model implemented by teachers in selected schools and the Core model plus is a combination of Core model activities and parental component. The parental component includes story telling for children, community mini-libraries and asking parents to regularly read for their children among others. Baseline survey was undertaken between the months of July and August 2009 for grades 1 and 2 of 2009 and in the months of February and March for grade 1 of 2010. The endline survey was undertaken from the end of June to July 2011 in all the grades for which baseline data had been collected.
The impact evaluation was designed to answer the following research questions: Are children in lower primary grades (1, 2 and 3) able to read and do mathematics calculations more proficiently as a result of the Reading to Learn/scaffolding approach?; what are the differences in proficiency for children who have been exposed to parental involvement in the Reading to Learn Approach (core model plus) compared to those exposed to the Reading to Learn Approach with no parental involvement (core model), and compared to control schools?; and what are the key contributing factors to these improvements in numeracy and literacy in grades 1, 2 and 3? The evaluation also aimed to find out the cost-effectiveness of the core model and core model plus.
To assess the impact of EAQEL on numeracy and literacy in early grades, a randomized controlled trial (RCT) design was adapted. The adoption of this design followed extensive consultation between APHRC as impact evaluators and AKF as intervention implementers. It was agreed that the benefit of an RCT design, particularly its simplicity in interpreting the results and ability to clearly isolate the impact of the intervention through the control group counterfactual, while at the same time avoiding selection bias problems that can exist in other evaluation designs, was powerful. The superiority of RCTs over other evaluation method also include clear results, elimination of lengthy caveats, and the possibility of future meta-analysis (Hutchinson and Styles, 2010, p7). In addition to this quantitative approach of an RCT, focus group discussions were conducted with parents to provide insights to the intervention and its implementation experiences.
To minimize contamination, randomization was done at cluster level. These were pre-existing clusters of schools determined by AKF's administrative units, in both Kenya and Uganda. The experimental sample consisted of 41 “clusters” of schools. In Kenya, the clusters were groups of schools determined by AKF that contained 1 to 8 geographically proximate schools. In Uganda, the clusters were administratively determined sub-counties that contained 2 to 16 schools each. In total there were 229 schools in 41 clusters (Kenya having 31 clusters-120 schools and Uganda 10 clusters-109 schools).
Kenya's clusters are divided between two districts, Kwale and Kinango. The district of Kinango was further subdivided into clusters that did, or did not, participate in the Kenya School Improvement Program (KENSIP) intervention. KENSIP was an earlier intervention undertaken by AKF whose effect needed to be isolated from the effect of EAQEL. Uganda's clusters were also divided between two districts, Amolatar and Dokolo. The final randomization occurred within 5 strata (defined by 3 districts, plus one district divided between KENSIP and non KENSIP). Of the 41 clusters, 19 received the treatment (either Core or Core Plus, depending on the district) and 22 were in the control group. In general, all schools residing within treatment clusters received the treatment, while control schools did not. However, one school in Amolatar and one in Dokolo were randomly assigned to a control cluster, but were later selected to be “model treatment schools” by AKF (a classic instance of experimental crossover between treatment and control conditions).
A total of 120 and 109 schools participated in Kenya and Uganda, respectively. In each grade, a random sample of 20 pupils was selected taking into account the proportion of girls and boys in the class. The sample of pupils was increased to 25 for the 2010 grade 1 in order to cater for any possible attrition due to absenteeism and school transfers. The same pupils were followed at the endline survey that took place between June and July 2011. To address the attrition problem at endline, the pupils who couldn't be traced were randomly replaced taking sex into consideration. At endline survey, 13,944 pupils participated in the evaluation, with 67.4% being the follow-up group traced from the baseline. In total 445 teachers were interviewed, and 12 FGD's were conducted in both control and treatment schools in the districts where core model plus was implemented. To undertake the impact evaluation, several tools were developed including pupil assessment tools - two for literacy and one for numeracy, teacher characteristics questionnaire, classroom observation checklist, school characteristics questionnaire, household characteristics questionnaire and the focus group discussion (FGD) guide.
