Abstract
This was an external evaluation of the Tayari pre-primary school programme. Tayari is an early childhood development and education (ECDE) intervention in Kenya funded by the Children's Investment Fund Foundation (CIFF). The intervention is implemented by the RTI International, in partnership with the Kenya Ministry of Education (MoE), and evaluated by the African Population and Health Research Centre (APHRC). The programme, which ran from November 2014 to July 2018, aimed to develop a cost-effective, scalable model of ECDE that would ensure that children who are prepared to join primary grade 1 are cognitively, physically, socially and emotionally ready to start, and succeed in primary school. The programme focused on improving learners' knowledge and skills in literacy, numeracy, health and hygiene as well as psychosocial skills; it targeted pre-primary school children in ECDE centres in four counties in Kenya: Laikipia, Nairobi, Siaya and Uasin Gishu. Both public and Alternative Provision of Basic Education and Training (APBET) ECDE centres were targeted in Nairobi while only public centres were targeted in the other three counties. As the programme's external evaluator, APHRC's role in Tayari was mainly to assess the impact of the programme on preparing children for primary school; and to assess the cost-effectiveness of the programme.
The evaluation, which adopted a randomized control trial (RCT) design, involved three separate treatment arms and one control arm for each type (public and APBET) of ECDE centre or school. The first treatment arm (T1) received two components of the intervention - teacher training and classroom instructional support; the second treatment arm (T2) received the two components in the first treatment plus an instructional materials component (consisting of learner workbooks, teachers' guides, and other instructional materials); the third treatment arm (T3) received all the three components in the second treatment arm, plus a health component. The control arm received no treatment but would receive the components proven to work after completion of the piloting phase in 2018.
In order to examine the effects of duration of exposure to the Tayari intervention, and because of limitation of funds, half the number of schools required to detect the required effect size were added in the study in 2016 (referred to as “Phase 1” schools), and the other half was added in 2017 (referred to as “Phase 2” schools) - meaning that the 2017 endline study covered in this report had a full sample needed to detect the stipulated mean effect size based on study estimates. This also means that Phase 1 schools were exposed to the intervention for two years while Phase 2 schools were exposed for one year, before the end term evaluation was conducted.
In the paragraphs below, we provide a summary of the answers to the key evaluation questions. Detailed answers to these questions are provided in the final chapter of this report.
Research question 1: What is the impact of (i) classroom instruction model (T1&2), and (ii) classroom instruction combined with health component (T3) on learner achievement?
The two packages involving teacher training and classroom instruction components (also referred to as “classroom instructional model”) plus an instructional materials component - T2 and T3 - improved overall school mean scores by 0.34 and 0.31 standard deviations respectively in public ECDE centres, and by 0.52 and 0.42 standard deviations respectively in APBET centres. On the other hand, the T1 package, which involved the classroom instructional model only (that is, without the instructional materials component), improved school mean scores by 0.30 and 0.08 standard deviations in public and APBET centres respectively.
Research question 2: Does the effect of the interventions vary by public versus APBET, length of exposure to the intervention and learner sex?
a) Overall, the magnitude of the impact of the intervention was bigger in APBET schools that were in T2 and T3 than in public schools. The impact was least felt in T1 APBET.
b) The impact of the intervention on school mean scores was practically significant even after stratifying by phase. However, the impact of the intervention was of greater significance among Phase 2 schools than among Phase 1 schools, which seemed to contradict the notion that longevity of exposure to the intervention would have an additional advantage. However, it is reasonable to argue that the lower impact on Phase 1 schools may be due to teething problems at the start of the programmme and exposure to change - teachers were encouraged to operate outside their comfort zones.
c) The intervention did not seem to impact differently by learner sex in APBET schools. However, in public schools, there seem to be some better results for girls than for boys, especially in T1 and T3.
Research question 3: Are health interventions together with classroom instruction models more effective in improving learning outcomes than classroom instruction model alone?
The impact of the packages involving classroom instructional model and provision of instructional materials plus the health component (T3) on school mean scores did not differ much from that of the package involving classroom instructional model and provision of instruction materials only (T2). Hence, the health component did not seem to offer additional advantage in terms of improved school readiness overall score.
Research question 4: Are interventions cost-effective? What are the costs of the interventions and their incremental effects on assessment scores?
Research question 5: Which aspects of Tayari worked well, and what didn't?
By and large, this question pertains to process evaluation and thus it is handled in a separate report. Nevertheless, based on the quantitative analyses carried out in this report, it would seem that the teacher training and classroom instructional support package (T1) worked better if accompanied by provision of instructional materials (T2).
To get children ready to transition to primary school, the Tayari programme seeks to strengthen the existing ECDE model in Kenya through the following four key intervention components:
i. Training for DICECE officers (supporting public centres) and instructional coaches (supporting APBET centres) in the use of tablet-based technology to supervise and mentor ECDE teachers in improved pedagogy approach;
ii. Teacher training component to increase active learning and instructional time. This component also focused on development of child-centred instructional materials (eg. charts, flashcards, counters and other materials developed by teachers using low cost locally available resources), and utilization of books and teachers' guides;
iii. Books and teachers' guide (also referred to as “instructional materials”) component involves providing each learner with low-cost instructional materials (workbooks) on a 1:1 ratio. The teachers' guides developed through the Tayari programme are aligned to the national curriculum and are linked to the learning materials, which contain activities that are matched to the lessons. The teachers' guides further facilitate the teaching of the official ECDE curriculum developed by the Kenya Institute of Curriculum Development (KICD);
iv. Health support component that integrates health and hygiene to support the holistic development of the child. Health support is provided to ECDE centres by Community Health Assistants/Volunteers (CHA/Vs) to improve key health and hygiene aspects such as hand washing, latrine use, and point of use water treatment. This component also includes strengthening the documentation and use of child health data for decision making. It was anticipated that this component would reduce the frequency of illnesses among learners, and thus, improve school attendance by reducing absenteeism caused by health related issues. Evaluation of this component involves determining improved school readiness score for health-support exposed ECDE centres compared to the control group and T2 group.
Tayari implementation
The Tayari intervention components were implemented in selected public and APBET ECDE centres within each of the four targeted counties through the three treatment packages described below:
a) Treatment 1 (T1) intervention arm - or T1 treatment package - schools receive two components (i) DICECE training and (ii) teacher training and support. Teachers are supported to use existing ECDE instructional materials and develop their own materials. The T1 package is also referred to as the “classroom instructional” package in this report.
b) Treatment 2 (T2) intervention arm schools receive the classroom instructional components as in (a) above, plus the instructional materials component (which includes teachers' guides, learner workbooks, and other materials).
c) Treatment 3 (T3) intervention arm schools receive the components in (a) and (b) packages above, in addition to health support component.
