East African Quality in Early Learning (EAQEL) 2009-2011
This is an impact assessment report of an independent evaluation of the Aga Khan Foundation's (AKF) East African Quality in Early Learning (EAQEL) initiative to determine whether the initiative improves learning outcomes in the early grades (1-3) in two districts in Kenya and two districts in Uganda as was intended. The initiative is also referred to as the Reading to Learn (RtL) approach. The districts covered by the study are Kwale and Kinango in Kenya and Amolatar and Dokolo in Uganda. The four districts were selected by AKF because they were consistently performing poorly in the national examinations in both countries. The EAQEL initiative has two components: Core model and Core model plus. The Core model involved a 'new' instructional model implemented by teachers in selected schools and the Core model plus is a combination of Core model activities and parental component. The parental component includes story telling for children, community mini-libraries and asking parents to regularly read for their children among others. Baseline survey was undertaken between the months of July and August 2009 for grades 1 and 2 of 2009 and in the months of February and March for grade 1 of 2010. The endline survey was undertaken from the end of June to July 2011 in all the grades for which baseline data had been collected.
The impact evaluation was designed to answer the following research questions: Are children in lower primary grades (1, 2 and 3) able to read and do mathematics calculations more proficiently as a result of the Reading to Learn/scaffolding approach?; what are the differences in proficiency for children who have been exposed to parental involvement in the Reading to Learn Approach (core model plus) compared to those exposed to the Reading to Learn Approach with no parental involvement (core model), and compared to control schools?; and what are the key contributing factors to these improvements in numeracy and literacy in grades 1, 2 and 3? The evaluation also aimed to find out the cost-effectiveness of the core model and core model plus.
To assess the impact of EAQEL on numeracy and literacy in early grades, a randomized controlled trial (RCT) design was adapted. The adoption of this design followed extensive consultation between APHRC as impact evaluators and AKF as intervention implementers. It was agreed that the benefit of an RCT design, particularly its simplicity in interpreting the results and ability to clearly isolate the impact of the intervention through the control group counterfactual, while at the same time avoiding selection bias problems that can exist in other evaluation designs, was powerful. The superiority of RCTs over other evaluation method also include clear results, elimination of lengthy caveats, and the possibility of future meta-analysis (Hutchinson and Styles, 2010, p7). In addition to this quantitative approach of an RCT, focus group discussions were conducted with parents to provide insights to the intervention and its implementation experiences.
To minimize contamination, randomization was done at cluster level. These were pre-existing clusters of schools determined by AKF's administrative units, in both Kenya and Uganda. The experimental sample consisted of 41 “clusters” of schools. In Kenya, the clusters were groups of schools determined by AKF that contained 1 to 8 geographically proximate schools. In Uganda, the clusters were administratively determined sub-counties that contained 2 to 16 schools each. In total there were 229 schools in 41 clusters (Kenya having 31 clusters-120 schools and Uganda 10 clusters-109 schools).
Kenya's clusters are divided between two districts, Kwale and Kinango. The district of Kinango was further subdivided into clusters that did, or did not, participate in the Kenya School Improvement Program (KENSIP) intervention. KENSIP was an earlier intervention undertaken by AKF whose effect needed to be isolated from the effect of EAQEL. Uganda's clusters were also divided between two districts, Amolatar and Dokolo. The final randomization occurred within 5 strata (defined by 3 districts, plus one district divided between KENSIP and non KENSIP). Of the 41 clusters, 19 received the treatment (either Core or Core Plus, depending on the district) and 22 were in the control group. In general, all schools residing within treatment clusters received the treatment, while control schools did not. However, one school in Amolatar and one in Dokolo were randomly assigned to a control cluster, but were later selected to be “model treatment schools” by AKF (a classic instance of experimental crossover between treatment and control conditions).
A total of 120 and 109 schools participated in Kenya and Uganda, respectively. In each grade, a random sample of 20 pupils was selected taking into account the proportion of girls and boys in the class. The sample of pupils was increased to 25 for the 2010 grade 1 in order to cater for any possible attrition due to absenteeism and school transfers. The same pupils were followed at the endline survey that took place between June and July 2011. To address the attrition problem at endline, the pupils who couldn't be traced were randomly replaced taking sex into consideration. At endline survey, 13,944 pupils participated in the evaluation, with 67.4% being the follow-up group traced from the baseline. In total 445 teachers were interviewed, and 12 FGD's were conducted in both control and treatment schools in the districts where core model plus was implemented. To undertake the impact evaluation, several tools were developed including pupil assessment tools - two for literacy and one for numeracy, teacher characteristics questionnaire, classroom observation checklist, school characteristics questionnaire, household characteristics questionnaire and the focus group discussion (FGD) guide.
The study collected data from schools in Kwale and Kinango districts in Kenya and Amolatar and Dokolo districts in Uganda
Unit of Analysis
Pupils and schools
This was in aintervention study and APHRC conducted an independent evaluation. Schools in the four partcipating districts on Kenya and Uganda were selected. In each school a sample of 25 pupils in grades 1 & 2 in 2009 was drawn and assessment tools administered. In 2010, a second baseline of the incoming grade 1 was conducted. The endline data collection was conducted in 2011
Producers and sponsors
Authoring entity/Primary investigators
African Population and Health Research Center
Dr. Moses Oketch
Dr. Moses Ngware
William and Flora Hewlett Foundation
Randomized Controlled Trial (RCT) impact evaluations attempt to answer the following question: what would have been the outcome if the participants did not receive treatment (Ravallion, 2001). In most impact evaluation studies, a comparison group is used as a counterfactual to establish what the outcome would have been. In randomization, the participants are randomly assigned to treatment and comparison groups. The underlying principle is that any observed average difference in outcome can only be as a result of the program or intervention (Duflo et al. 2006). The impact evaluation reported in this report used an RCT method. The benefit of an RCT design is the simplicity in result interpretation and the clear isolation of the impact of the intervention through the control group counterfactual, while at the same time avoiding selection bias problems that exist in other evaluation designs. In addition to the quantitative approach of RCT, focus group discussions (FGDs) were conducted with parents to provide explanations for the observed results. This was restricted to core model plus in order to highlight mechanisms by which parental involvement might influence the outcomes of the intervention.
SAMPLING OF SCHOOLS
Pre-existing AKF clusters of schools in the case of Kenya and sub-counties in the case of Uganda were randomly assigned to either the control or treatment group of the EAQEL intervention. The sub-counties in Uganda are effectively equivalent to Kenyan clusters, and we will refer to both using the cluster terminology for the remainder of the study. By this design, all schools residing within treatment clusters received the treatment, and all schools within control clusters did not. Through clustering, contamination which may have resulted if both treatment and control schools were in the same cluster was minimized.
There were a total of 41 clusters in the study, with 31 in Kenya and 10 in Uganda. Kenya's clusters fall within two districts, Kwale and Kinango. The district of Kinango was further subdivided into clusters that did, or did not participate in the Kenya School Improvement Program (KENSIP) intervention. KENSIP was an earlier intervention undertaken by AKF whose effect needed to be isolated from the effect of EAQEL. Uganda's clusters also fall within two districts, Amolatar and Dokolo. The final randomization occurred within 5 strata (defined by 3 districts, plus one district divided between KENSIP and non KENSIP). Of the 41 clusters, 19 received the treatment (either Core or Core Plus, depending on the district) and 22 were in the control group. However, one school in Amolatar and one in Dokolo were randomly assigned to a control cluster, but were later selected to be "model treatment schools" by AKF (a classic instance of experimental crossover between treatment and control conditions).
SELECTION OF PUPILS
The design of the study was such that it was not necessary (and financially feasible) to assess all pupils in each grade in the sampled schools. Therefore a random sample of 20 pupils was selected in each grade. The random sampling was done by first grouping pupils by sex; and then selecting each sex based on their proportion in the class. Based on the baseline I experience, the sample was increased to 25 pupils for the 2010 grade 1 in our baseline II in order to allow for any possible attrition due to absenteeism and school transfers. The same pupils were followed at the endline survey that took place between June and July 2011. During the endline survey, pupil absenteeism presented a sample attrition problem. To address the attrition problem at endline, the pupils who were lost to follow-up were randomly replaced taking sex into consideration. This did not pose any methodological threat to the study because the intervention was administered at class level.
Deviations from the Sample Design
100% at baseline and 67.4% of the pupils were successfully followed to endline, replacement of the lost was undertaken
Dates of Data Collection (YYYY/MM/DD)
Mode of data collection
The logistics on the group involved putting the field team into teams, headed by a team leader. Each site also had an overall supervisor. Several measures were taken to ensure that quality data were collected during the period. First, team members in various study sites held daily meetings to discuss issues related to fieldwork. Second, team leaders went through all the questionnaires checking for errors including inconsistencies. If a team leader found inconsistent information while editing, he/she had to go back to the same household to confirm the information with the person who responded to the questionnaire. All team members edited their work daily before submitting to their respective team leaders. Third, team leaders also accompanied different teams to observe data collection. They also had sit-ins with FIs and conducted random spot-checks to ensure quality. Finally, the core research team also visited every study site and conducted random spot checks. Following these visits, the core research team held meetings with the teams and communicated issues and challenges that they came across and brainstormed on ways to improve the quality of data collection. In general, close and intensive supervision by team leaders and researchers ensured that the survey was conducted in a professional way and that quality data was collected.
Type of Research Instrument
To undertake the impact evaluation, several tools were developed. They included the following:
1. Pupil assessment tools: one for each measure of Literacy and one for Numeracy
2. Teacher characteristics questionnaire
3. Classroom observation checklist
4. School characteristics questionnaire
5. Household characteristics questionnaire
6. Focus group discussion study protocol
1.1.1 Pupil Assessment Tools
The EAQEL teaching approach focuses on literacy and numeracy in early grades 1, 2 and 3. In developing standardized assessment tests to assess the impact of EAQEL, several consultation meetings with key stakeholders and experts were held. The stakeholders included EAQEL implementing agency (AKF), APHRC, national assessment experts, national curriculum experts, academics, and practitioners in numeracy and literacy assessment. These experts came from Kenya and Uganda as well as internationally. Several stages were involved in developing the assessment tools. First, a pool of questions was developed drawing on curriculum from both countries. For instance, in numeracy, the team came up with a pool of 50 test items in each of the grades. The competencies and skills for grade 1 were examined and agreed upon by the team. In the case of grades 2 and 3 the competency and skills domains were repeated but the level of difficulty of test items required higher order thinking.
Second, the pool was refined with the input of assessment experts and from this pool the final test items were selected. Third, the test items were translated into both Kiswahili and Lang'o, which are the languages of instruction in the Kenya and Uganda study sites respectively. These languages are also widely spoken in the catchments area of these schools.
There was one test tool for numeracy and another two test tools for each of written and oral literacy for all the three grades with students in higher grades taking all of the lower grade questions along with the questions appropriate for their grade. The rationale for having one assessment tool covering the three grades was to permit determination of how pupils in higher grades scored on items for lower grades (for instance to determine how competent grade 2 pupils would be on grade 1 items in both numeracy and literacy).
1.1.2 School and Household characteristics questionnaire
Other instruments that were developed by APHRC and agreed on by both partners included questionnaires to gather information on the schools, teacher's characteristics and household characteristics as well as classroom resources and teacher preparedness. These were adapted from ongoing APHRC research work that collects similar information. The purpose of the observation checklist was to determine whether treatment and control schools in the study were similar in terms of the availability of such resources. At the endline, the observation checklist was instrumental in capturing the resource contribution made by EAQEL to treatment classrooms and which could be linked to the impact outcome.
Data editing took place at a number of stages throughout the processing, including:
a) Office editing and coding
b) During data entry
c) Structure checking and completeness
d) Secondary editing
e) Double entry
During data entry, any issues or incosistencies arising from the questionniares were edited in coslulation with the field team and sometimes with the respondents.
Estimates of Sampling Error
Director of Research
African Population and Health Research Center
All non-APHRC staff seeking to use data generated at the Center must obtain written approval to use the data from the Director of Research. This form is developed to assess applications for data use and facilitate responsible sharing of data with external partners/collaborators/researchers. By entering into this agreement, the undersigned agrees to use these data only for the purpose for which they were obtained and to abide by the conditions outlined below:
1. Data Ownership: The data remain the property of APHRC; any unauthorized reproduction and sharing of the data is strictly prohibited. The user will, therefore, not release nor permit others to use or release the data to any other person without the written authorization from the Center.
2. Purpose: The provided data must be used for the purpose specified in the Data Request Form; any other use not specified in the form must receive additional or separate authorization.
3. Respondent Identifiers: The Center is committed to protecting the identity of the respondents who provide information in its research. All analytical data sets (both qualitative and quantitative) released by the Data Unit MUST are stripped of respondent identifiers to protect the identity of the respondents. By accepting to use APHRC data, the user is pledging that he/she will not, under any circumstance, regenerate the identifiers or permit others to use the data to learn the identity of any individual, household or community included in any data set.
4. Confidentiality pledge: The user will not use nor permit others to use the data to report any information in the data sets that could identify, directly or by inference, individuals or households.
5. Reporting of errors or inconsistencies: The user will promptly notify the Head of the Statistics and Survey Unit any errors discovered in the data as soon as the errors are discovered.
6. Publications resulting from APHRC data: The Center requires external collaborators to work with APHRC staff on all publications resulting from its data. In order to facilitate this, lead authors should send a detailed concept note of the paper (including the background, rationale, data, analytical methods, and preliminary findings) to the Principle Investigator (or Theme Leader) for the project (with a copy to the Director of Research), who will circulate the abstract to concerned researchers for possible expression of interest in participating in the publication as co-authors. Any exception to the involvement of APHRC staff should be approved by the Director of Research, APHRC.
7. Security: The user will take responsibility for the security of the data by ensuring that the data are used and stored in a secure environment where access is password protected. This will ensure that non-authorized people should not have access to the data.
8. Loss of privilege to use data: In the event that APHRC determines that the data user is in violation of the conditions for using the data, or if the user wishes to cancel this agreement, the user will destroy the data files provided to him/her. APHRC retains the right to revoke this agreement or informs publishers to withhold publication of any work based wholly or in part on its data if the conditions for using the data are violated.
9. Acknowledgement: Any work/reports from this data must acknowledge APHRC as the source of these data. For example, the suggested acknowledgement for NUHDSS data is:
"This research uses livelihoods data collected under the longitudinal Nairobi Urban Health and Demographic Surveillance System (NUHDSS) since 2006. The NUHDSS is carried out by the African Population and Health Research Center in two slums settlements (Korogocho and Viwandani) in Nairobi City."
Additionally all funders, the study communities that provided the data, and staff who collected and analyzed or processed the data should be acknowledged.
10. Deposit of Reports/Papers: The user should submit electronic and paper copies of all publications generated using APHRC data to the Policy Engagement and Communications Department, with copies to the Director of Research.
11. Change of contact details: The user will promptly inform the Director of Research of any change in your personal details as contained on this data request form.
African Population and Health Research Center (APHRC), East African Quality in Early Learning (EAQEL) 2009-2011 Version 1.1 June 2017. doi:10.20369/aphrc-041:2003.1.01
Disclaimer and copyrights
The user of the data acknowledges that the original collector of the data, the authorized distributor of the data, and the relevant funding agency bear no responsibility for use of the data or for interpretations or inferences based upon such uses.