In this post, I discuss what we have learnt from 20 years of
randomised evaluations. I argue that effective interventions (i.e. ones that improve
attendance or exam scores) typically do two things well: first, they narrow
their focus onto a specific malfunction that is inhibiting education outcomes;
second, they do so in a way that properly accounts for the incentives of the
agents studied.
What Works
Conditional Cash Transfers
Conditional cash transfers (CCTs) provide a good example of
this. CCTs are remarkably simple: participants in the treatment group receive a
direct cash transfer (usually ~$50 per month) if their child attains a certain
level of attendance in school. CCTs work well as they directly incentivise
parents to send their children to school while simultaneously relieving a
financial constraint that may have previously prevented them from doing so. In
a review article, Glewwe and Muralidharan (2015) outline 27 RCTs to test CCTs,
24 of which produce a statistically significant increase in school attendance.
As argued in my last post, the unanimity of these results in a variety of
contexts makes the effectiveness of CCTs an extremely robust finding. Of
course, CCTs cost money – but the financing could theoretically be provided by
donor funding, particularly if it is redirected away from other forms of
budgetary support with less proven results.
Teacher Performance Pay
Another good example is the use of teacher performance pay.
Again, performance pay seeks to address a specific malfunction (demotivated
teachers) in a way that is appreciative of their incentives. In the same review
article, Glewwe and Muralidharan cite 5 high-quality RCTs, all of which show
that exam results significantly improve in treatment schools where teachers are
paid according to performance. Not only this: Kremer et al. (2005) show that
there is a significant positive correlation between teacher performance and the
extent of performance pay desired beforehand, suggesting that good teachers
‘know who they are’, implying additional sorting benefits from performance pay
(i.e. better quality teachers are attracted in the first place).
What Doesn’t
Discretionary Rewards
However, performance-based interventions fail when insufficient
attention is paid to incentives. A good example of this is a study by Kremer
and Chen (2001), which gave Kenyan headteachers discretion to award bicycles to
pre-school teachers based on good attendance. Somewhat predictably, every teacher
was reported to have good attendance and received a bicycle. However, when
absence was externally audited, attendance had not changed at all from
baseline. The study did not take into account the headmaster’s incentive to collude
with his fellow teachers.
CCT programmes must also be mindful of incentives. Suppose
you are a farmer with four children who help out with farm work when required.
An NGO comes along and offers you money if you send two of your kids to school
– how might you respond? A sensible way might be to remove your non-sponsored
kids from school completely, to compensate for the shortfall of labour on the
farm. Several studies have documented these types of ‘negative spillovers’ to
siblings (e.g. Barrera-Osorio et al., 2011); interventions must be wary of
these and redesigned accordingly.
More-of-the-same Inputs
Interventions may also fail when inadequate care is taken to
understand the fundamental barriers to learning. For example, when it comes to
improving test scores, ‘more-of-the-same’ inputs (e.g. more textbooks,
stationary and teachers) have proven remarkably ineffective (Kremer et al.,
2013). Buying more textbooks may sound sensible, but when children are
illiterate, unsupervised and of wildly varying abilities, these textbooks are
unlikely to do anything but gather dust. These examples should caution us
against ‘lazy’ interventions that throw more of the same ingredients at a
problem and expect different results. We need to be more creative. A study by
Banerjee et al. (2007) sets a good example: to deal with the problem of ‘left
behind’ children, the authors implement and test two novel remedial education
programmes. The first hired young women from the community to teach basic literacy
and numeracy skills; the second consisted of a computer-assisted learning
programme tailored to the child’s ability. On average, the two interventions
increased test scores by 0.28 and 0.47 standard deviations respectively.[1]
The author’s estimate that extending this programme would be 12-16 times more
cost effective than simply hiring more teachers.
The Issue of Scalability
While these are undoubtedly promising, there are numerous
difficulties associated with scaling up these interventions to form meaningful
policy change. For instance, RCT-based interventions may change market prices
or produce negative spillovers when scaled up, both of which may shift the net
benefit of the programme or the identities of the beneficiaries. On a more
practical level, wide-scale policy change often requires immense political will
and capacity – neither of which are required for a successful RCT (Banerjee et
al., 2017).
However, these issues are not insurmountable. The experience
of Pratham, an Indian NGO, gives cause for hope. In 2004, Pratham ran a
successful RCT which grouped children based by ability rather than ages, helping
children that had previously been left behind. The Pratham-approach was then
integrated into some pilot schools. The intervention had to be adapted to be
both feasible and financially viable: several variations were retested by
further RCTs to determine the most effective. In the end, the best variation
was the adding of a ‘Pratham-specific’ hour to the school day. Today, the
additional-hour model exists in 107,921 schools across 13 states of India,
reaching nearly five million schoolchildren. Five million children are now
being taught daily by a strategy proven to help them learn – the welfare
implications of this are staggering. These types of interventions make me
excited to work in the education sector, and help convince me that RCTs can be
a powerful tool to improve the world we live in.
[998 words]
References
Glewwe and Muralidharan (2015) have a very good review
article which outlines the education RCTs that have worked vs. those that
haven’t. Banerjee et al. (2017) provide one of the best and most up-to-date
discussions of the challenges associated with scaling up successful RCTs to
policy change. All cited articles are listed below:
·
Banerjee et al. (2017), ‘From Proof of Concept
to Scalable Policies: Challenges and Solutions, with Application’
·
Banerjee et al. (2007), ‘Remedying Education:
Evidence from Two Randomised Experiments in India’
·
Barrera-Osorio et al. (2011), ‘Improving the
Design of Conditional Transfer Programs: Evidence from a Randomised Education
Experiment in Colombia’
·
Glewwe and Muralidharan (2015), ‘Improving
School Education Outcomes in Developing Countries: Evidence, Knowledge Gaps,
and Policy Implications’
·
Kremer et al. (2005), ‘Teacher Absence in India:
A Snapshot’
·
Kremer and Chen (2001), ‘Interim Report on a
Teacher Attendance Incentive Program in Kenya’
[1]
Standard deviation is a measure of how spread out numbers are. Different
contexts have very different data distributions and averages; reporting results
in standard deviations is a way to ‘standardise’ this so we can easily compare
across contexts
No comments:
Post a Comment