Children in their classroom in El Renacimiento school, in Villa Nueva, Guatemala. Photo: Maria Fleischmann / World Bank
Key messages from a roundtable discussion in London, 17 November 2014
This summary was submitted by Pauline Rose, Ed Barnett, Rob Whitby, Steve Packer and Susan Nicolai.A roundtable discussion co-hosted by DFID, University of Cambridge, ODI and UKFIET* brought together technical experts to identify recommendations for indicators to track learning and equity post-2015. The discussions were positioned in the context of the Technical Advisory Group (TAG) to the Education For All Steering Committee (see consultation on education indicators) and the recently-released UN Data Revolution Report. The roundtable included over 40 participants from UK universities, think tanks, European donor agencies and NGOs. This note summarises the key messages from the day but does not attempt to fully represent the diversity of opinions shared.
Common themes throughout the discussions included:
Download all materials from 17 November Indicators Roundtable.
- indicators are important to hold policymakers nationally and globally to account for agreed targets.
- recognition that the selection of indicators is both a technical and political process, but that ultimately indicators need to be technically-robust to be useful.
- the need to make sure concepts in the proposed targets are clearly defined for them to be measurable.
- the need to take a short-/medium-/and longer-term perspective to what can be measured.
- the importance of cooperation between agencies involved in data collection from an early stage.
- the need for education experts to engage with the broader data revolution process: to share, inform and learn from this process which is likely to be central to post-2015 monitoring.
Measuring learning and skills post-2015
Presentations reflected on the measurement of learning, focusing on Muscat target 2 and OWG target 4.1. A proposal for a common learning metric was presented and discussed. Lessons from measuring learning in the UK context were also discussed. The TAG document proposes that the aim should be for technically robust and globally comparable indicators whose measures are (1) valid and reliable across all countries and (2) assessed by a similar question or item. However, participants noted that, in reality, the measure of success should be “fitness for purpose” and not “perfection”.
Key messages on learning
- Measuring progress towards a global learning target can and should be based primarily on national valid and reliable assessment data.
- The primary purpose of such national assessments should be to inform policy, planning and practice within countries. Such assessments are also not a substitute for formative classroom assessments, which are vital for improving learning.
- Work on the common metric, drawing where possible on existing assessments, should be prioritised and efforts made to publicise and get feedback on the proposal. The common metric proposal should be made available online alongside the existing TAG consultation document.
- Learning from the UK: high-stakes learning assessments can lead to grade inflation, teaching to the test and a loss in credibility of the assessment as an objective measure of learning. There are benefits in having a combination of such assessments along side international ones (such as PISA) to allow triangulation, and so overcome manipulation that can happen with either type of assessment on their own.
- Equity must be a central concern in all efforts to track learning outcomes. This requires consideration of the age or grade of assessment, given that learning inequalities start at an early age. In the UK, for example, inequalities are so entrenched by secondary schooling that policies at this stage have had limited impact.
- To overcome inequality, priority should be given to indicators that are comparable across countries, in particular basic reading in the language of instruction and numeracy skills. These are important both in their own right, and as a means to learning other skills.
- Learning should be tracked globally at an age by which children would be expected to have spent some time in school (for example equivalent to grade 3), and by the time they should have completed primary school.
Key messages on skills
- Beyond literacy and numeracy, one possible area to focus on for cross-country comparability more immediately could be ICT skills. It is unlikely to be possible to measure “non-cognitive” skills in an internationally comparative manner in the short term, but certain measures of non-cognitive skills could be measured in specific national contexts.
- The OECD PIAAC and World Bank STEP routes are a good starting point for measuring skills comparatively, and should be adapted to be shorter, cheaper, and more suitable for developing country contexts.
- The focus of measuring skills should be on adults of all ages, not just the young.
- Another approach could be to collect data on policy areas, such as whether a functional national qualification system exists. The advantage of this is that governments are taking skills seriously if they have such a system. The disadvantage is that the European qualification system has been criticised, owing to covering a too-narrow approach to skills.
Measuring equity post-2015
The session started by highlighting the importance of putting equity and leaving no one behind at the heart of post-2015 goals – given this has been one of the greatest failures of the MDG process. A ‘stepping stones’ approach was proposed, which entails setting interim targets to make sure no one is left behind. A focus was then given to tracking the progress of people with disabilities who are often invisible in data, and thus overlooked by policymakers. While UNICEF and the Washington Group initiatives offer an immediate means to improve data collection on access to education for people with disabilities, approaches to measuring their learning might need to be longer-term.
Key messages
- Tracking progress to achieve equity should be integral to all targets, regardless of whether equity is also identified as a separate target.
- To ensure no one is left behind, it is important to track progress for disadvantaged groups in absolute terms, as well as to track the gap between them and advantaged groups.
- A ‘stepping-stones approach’ should be prioritised, including setting of interim targets.
- To accompany a stepping-stones approach, ‘quick-wins’ for policymakers should be identified so that progress can be established early-on, but with a longer-term planning horizon to make sure progress does not stall once low-hanging fruit have been captured.
- There is an urgent need to develop a globally-comparable measure of socio-economic status, given that income is not easily captured in all contexts. This is not only a concern for measuring progress towards education, so should be undertaken with those developing indicators for other targets.
- Household surveys are essential for tracking progress on inequality across time, so it is vital to improve their coverage and make sure they are undertaken at regular intervals. Surveys have most traction if they are designed to inform policy within countries, but with potential for greater use for comparability across countries.
- Existing household surveys track gender, rural/urban and, in some cases, income/wealth. It is important to recognise, however, that household surveys currently miss out important population groups (e.g. children with disabilities in institutions, nomadic population groups, those living in informal settlements in urban slums). Solutions are needed to collect data on these groups.
- Ideally, data from both administrative and survey data should be linked. In doing so, there is a need to be able to explain differences across different data collection instruments (whether due to sampling; questions etc.). The absence of recent census data is a major problem for some countries.
- There is a need to strengthen the inclusion of relevant education questions in wider (cross-country) household surveys, such as DHS and MICS.
These are great additions to the global discussion. I strongly sympathize/endorse. Thanks for taking the trouble to do this and report on it.
A couple of extra things I’d add:
a. I wonder whether one could recommend to UIS that one analyze all the various indicators in terms of which are closer to a sort of “tipping point” — that is, indicators which with just a little bit more effort could be tipped over into becoming valid, reliable, useful. Of those, which are also most important? So, to put it in a common but clichéd fashion: What is the lowest-hanging fruit that is also most nutritious? Seems to me that one has to admit that some of the indicators will be harder to come to agreement on (both the how one actually defines things and how one measures), harder to do pilot studies for, etc., than others. This seems to me should be no cause for shame or despair. It does not mean to neglect the fruit that’s hanging higher–it means we first have to plan to get a ladder, to go along with the clichéd analogy. Note that the discussion probably had this as an implicit theme in the dot point on short/medium/longer term… but it might be good to be explicit about the fact that some indicators really are pretty close to a tipping point, even if not quite there yet. Those, if they pertain to important issues, could be prioritized. (One would have to decide which ones they are.)
b. In prioritizing which indicators and how to measure and collect, we should perhaps keep the school and the teacher in mind, and design the indicators so that, if not the indicator per se, or the actual measurement tool, but some analog or “dual,” is useful at the classroom level. It is a separate agenda, and perhaps not suitable for a discussion on indicators, to propose that the development partners also invest in demonstrating just how to use indicators for classroom level improvement. This is probably the weakest point in all this. But, again, it may be a bit “unfair” to sort of insist that this be included in the indicator discussion. But the choice of indicators to prioritize should at least have this concern as part of the background.
Thanks for your comments Luis.
The next and urgent task of the Technical Advisory Group (headed by UIS) is to prioritise and rationalise the proposed indicator set. Your first point suggests a useful rationale as to how it might go about this. It would be good to hear your views on which indicators are examples of the “low-hanging fruit” you describe.
Top of this list, for me, is the work on measurement of foundation skills, particularly reading but also mathematics. This also links to the important work being done by UIS and its partners to develop a common learning metric, allowing different assessment approaches to map onto one common scale. This means national governments can continue to adopt an approach to assessment of their choosing but would have the additional option of being able to report against globally agreed benchmarks.
But as your second point alludes, global indicators and measurement frameworks are worth little unless they translate into improved practice in classrooms.
These are really helpful and pertinent comments, Luis. The first is a good suggestion. On the second, I wonder if we need to think differently about the choice of indicator depending on whether it is being used for national (and, by extension, global) planning and accountability versus those that are used directly to improve classroom practice – so in the hands of the teacher etc. The reason for this is that, in some ways, approaches like ASER/UWEZO are particularly helpful in being undertaken at the household level as, not only does this mean it captures learning of children whether in school or not, but also means that there is less likelihood of distortion of assessment in terms of teaching to the test etc. But this does not mean that other forms of assessment for classrom/school use isn’t also valuable.
Pauline
Let me comment on both.
There are a lot of indicators in the UIS-led framework–I count 62. So, if I had to prioritize according to which are both a) really useful because they are about issues that could affect huge numbers of children and youth, and b) could be made ready with only a bit more work, I’d probably say–these are all indicators already in the proposed framework:
As a general point, I’d say that two areas that are really important and there is already enough work so that one could make the work “jell” if there was enough leadership, are the two most important “end points” or “articulation/transition points” of the ed system: what I’d call “early years” (pre-school and the first couple of grades) and the “school to work and life” transition.
a. Closer agreement and testing of indicators of school readiness–relevant to the pre-primary agenda
b. More agreement and piloting of indicators for skills in the first few grades–could be part of the common learning metric work
c. Same with work-readiness skills, in particular the so-called non-cognitive ones
d. Agreement and methods on how to measure “youth pursuing nothing” (obviously the real interest is the complement)
e. Agreement on how to track equity issues
f. How to track whether the countries engage in purposeful pro-poor targeting of resources
More challenging but perhaps really important:
a. Development and agreement on a combined index of school quality (to make sure process and inputs are also taken care of)
b. Since there is so much research showing that well-supported teachers are perhaps THE key input into quality of education, agreeing on some measure or index of this factor.
c. My point f above may belong in this list more than in the previous one–not sure–but I think it is pretty important.
On the latter two I think there is less work already done. On the previous 5 I think there is enough work floating “out there” that it is more a matter of convening agreement. More actual work is needed (psychometric, field experimentation with the measurement) but at least there is good (though disparate) work that has gone on already…
The 2nd issue raised by you, and the 2nd issue raised by Pauline I think are not so much a matter of developing the metrics/indicators and then having an agreement on the indicators and how to measure them. They are sort of only “extensions” of the indicators issues–but should not be forgotten. I guess I would not become too mechanical about it. I don’t think it is a matter of having an indicator that itself is useful at the classroom level AND useful for reporting. It is more a matter of “simply” (!) remembering that for at least some of the key indicators, one needs some version or analog that is useful at the classroom and school level. I think tools such as ASER already get used that way and this is one indicator (or one metric) that is immediately useful both for reporting and for classroom/school work. But the metrics and indicators need not be THAT aligned–just somewhat aligned. In countries where learning outcomes are taken more or less seriously, and where teachers are relatively professional, it seems to me that teachers themselves often develop these analogs for use in the classroom via district-wide teacher groups and so on. That kind of thing needs to be supported. And it can reduce the tendency to over-focus on one final result and make things too high-stakes… (the distortions Pauline mentions)…
Quality education is must for children of every age group. Many people try to go with tutoring services for their children for a variety of reasons.