This blog was written by Keith M Lewin, Emeritus Professor of International Development and Education, University of Sussex, as a response to the GEEAP ‘Cost Effective Approaches to Improve Global Learning’ report, published in October 2020.
The Global Education Evidence Advisory Panel (GEEAP) has launched its report on ‘Cost Effective Approaches to Improve Global Learning’. Readers of the GEEAP report will want to compare and cross reference its findings on achievement with those already in the public domain and widely discussed at UKFIET Conferences in the past, to see how the recommendations resonate. They will also be interested to understand how this initiative maps onto the architecture of aid to education, which should continue to develop. Are these best buys already part of this architecture or have they been persistently over-looked? Do they now have to be grafted on to existing projects? What is the appetite from governments to adopt these good and bad buys. Will they continue to buy with their own money after the external finance stops? The report presents a taxonomy of interventions that are grouped under five headings.
Key issues for the GEEAP are: who is buying what, from whom, who is the Principal and who is the Agent in the transactions? What are governments, NGOs and donors buying and selling? Is there really a market that determines price, or is it a simply a framework for contracting services? Is a ‘smart buy’ the one that is best for the seller or for the buyer? If the best buy is also the best sale (e.g. performance related development bonds?), is everyone in the market ‘smart’? Are smart buys those where costs are very low relative to small benefits, or where benefits are very large relative to substantial costs? Which bargains are value for money but unaffordable? The question is how to avoid the collection and consumption trap that arises from seeking best buys for discrete educational interventions that are not part of a system specific development plan. If aid is to be directed towards the generic best buys as is planned, then it will lack local coherence with independently developed national plans.
The GEEAP report identifies one ‘great buy’, six ‘good buys’, three ‘promising options’ with “low” evidence, three ‘bad buys’, and five “Areas where governments need to make decisions or take action”. The last category includes investment in general in-service teacher training, the selection and deployment of teachers, and interventions related to gender, disability and violence though the evidence base for these is described as “low”.
Readers will make their own judgements as to how comprehensive and novel the lists are and whether the sampling and case studies can support global generalisations. In most of the categories, there are studies that show significant effects and other studies that indicate weaker or no impact of interventions. Averages may not be the best way to proceed. On a first read through, ten points stand out:
- The choice of buys is restricted to those interventions that are visible because they have been externally funded and have had RCT-based evaluations. Though the report claims it has considered a very wide range of interventions the final lists of buys that are presented are a small sub-set of the possibilities, and do not reflect the range of initiatives governments have taken without external financing. What you see is not all there is. The lists of impact in terms of Learning Adjusted Years of Schooling (LAYS) “omit interventions with a null effect” and appear to cover just 40 out of the 150 interventions in the main data set. There is therefore a confirmation bias in the sample towards interventions that have already been funded. There is a much wider range of potential good and bad ‘buys’ available than those identified. Many are excluded because they have no cost data.
- About 90 individual evaluation papers are cited and used to reach conclusions in the main report. More than half of those listed in the bibliography are from only five countries, three of which are mega federal states (India, Indonesia, Brazil), about 24% of are in South America and 25% in South Asia. Of the 34% in sub-Saharan African countries, 40% of these citations are from one country – Kenya . Strikingly, nearly 86% of studies are from lower and upper middle-income countries (LMICs 59% and UMICs 27%), and only 14% are from low-income countries (LICs). LICs are therefore under-represented and their needs may be different.
- The number of cases cited to arrive at ‘great buy’ and ‘good buys’ appears too small to lead to robust conclusions. Four of the seven great/best buys with cost benefit data are based on fewer than four country case studies each and these include cases where the effect is close to zero and where the interventions are different from each other in detail (see Figure 1 in the report). Out of the nine interventions with the greatest impact on LAYS, four are in Kenya, two are in India and two in Indonesia. None of these countries are LICs. In half the remaining cases, the gain calculated is less than one LAYS.
- The report only identifies one ‘great buy’, despite over 50 years of evaluations of development projects. This is “Giving information on education benefits, costs and quality”. This ranking is apparently based on a sample of only two country case studies with cost data (Figure 1, p32, N=2). The cost benefit is high because one case registers a huge gain (140 LAYS) and the other case no gain at all according to the chart. This intervention in Madagascar scores so highly because the costs are so small for providing information to large numbers school children and parents. This kind of intervention must also have diminishing returns – more and more information cannot keep increasing performance for all groups of students.
- There are only three ‘bad buys’ despite the global crisis in access and learning resulting from past investment. These bad buys include things that Ministries invest in, including “textbooks, technology hardware, uniforms, school grants, class size reductions, and general skill teacher training”. The implication is that external finance will not be available for bad buys unless they are associated with other unspecified inputs. The list will seem counter-intuitive to many aid recipients.
- Many studies do not have cost data on direct and indirect costs and do not account fully for donor costs. About “half of all interventions (that have cost data) produce no significant effects: these interventions are not included in the figures”. The reason for the omission is that “interventions with null impacts are not cost effective by definition”. Knowing which interventions have no effect is the only way of knowing what does not work. A verified list would be very helpful.
- Cost-effectiveness is measured by comparing the gains in LAYS per $100. In every real system, most of the $100 spent on any intervention that is sustained will be spent on teachers’ and other staff salaries and donor costs. The recurrent costs of sustainable interventions cannot be ‘bought’ but they can be financed in sustainable ways. Note also, learning gains are not linear, at least according to Piaget, Vygotsky and Bruner, so “a gain that would take a student in a high-quality system twice as long to achieve is one with twice the educational value” is also not a convincing metric. All gains are not equal cognitively, socially or economically.
- The average costs per child of schooling at primary level in LMICs are four times those of LICs. In UMICs they are eight times greater. If interventions are externally financed, what a dollar will buy will be very different between LICs, LMICs and UMICs. But the measured effect in terms of learning gain on standardised tests may be the same, so cost-effectiveness comparisons will not be very meaningful. A competition between countries to find the greatest gain at the least cost may not be the most efficient and effective way to allocate aid.
- There is no consolidated data on how many of the interventions cited are externally financed, how much of the total costs were domestically financed, how long ago they took place, how many are still running after external support ceased, and how many had effects beyond one cohort of students. Any robust list of ‘best buys’ would provide tables with this data. It is especially important where the intervention has been discontinued and is therefore unsustainable, and where “Hawthorne effects” create transient effects.
- COVID-19 will have extensive impacts on education budgets and will reshape cost-effectiveness, changing priorities based on pre-COVID-19 evaluations. Aid budgets are under pressure and bilateral commitment appear to have fallen by a third since the beginning of 2020, GDP in many countries will fall perhaps by more than 20%, and the poorest in the poorest countries will suffer most. Cost benefit will change as valuations reflect changing priorities and shifts in rates of return to investment. If the ‘great buy’ identified – information on benefits, costs and quality to parents and students – is re-purposed as a priority to reduce COVID-19 transmission and morbidity rates it might be an even greater buy.
© The World Bank
In conclusion, development projects (and rigorous systematic literature reviews), are now required to make their Theory of Change (ToC) explicit. GEEAP needs a ToC. If the learning crisis is really more of a persistent problem of financing and implementation than an inability to discover ‘what works’ it might be better to develop a Theory of Innovation (change with a purpose), applicable to open systems like education grounded in country level diagnostic studies on real systems. These would have to recognise the reasons why systems of learning and teaching persistently fail to deliver outcomes that meet the expectations of governments and donors. They should also make clearer who should ‘buy’ what from whom, and which interventions with long-term impact should be adopted, with or without the short-term incentive of external financing. Learning is much more than literacy and numeracy and effective education systems manage flows of students through a life cycle of organised learning experiences, not just a set of project-based interventions.
Target-setters need to be target-getters, especially when nudging systems with high inertia in new directions. More aid can be critical to catalysing reforms that make it more rather than less likely that sustainable educational development is the outcome. But the cost of interventions should not be confused with their value for development, or place in a medium-term strategy, that runs way beyond narrow indicators of achievement in basic education. The “best is not the enemy of the good” or the merely promising. The value of interventions is determined by their system specific context. Fashions change, sales always finish, and bargains often have a sting in the tail. In the new development decade strategies are needed that seek and value more differentiated visions of sustainable development. Development assistance is not an auction to get the best price, but a resilient, predictable and trustworthy partnership designed to achieve many different outcomes that depend on common values and commitments translated into actions.
More comments on the different options for effective aid and an indication of ways forward are available in an extended version of this paper at https://keithlewin.net/downloads
This blog was written as a response to the GEEAP ‘Cost Effective Approaches to Improve Global Learning’ report, published in October 2020. Other blogs written in response to the report are:
Smart buys: system reform, local-level buy-in and reaching the most marginalised
