Why are citizen led learning assessments not having an impact on home soil – and how can we change that?

Guest Blog by Colin Bangay on World Education Blog

Citizen-led learning assessments have been one of the most internationally influential educational initiatives of the decade. However, what of impact in their home countries?

This blog is written on ASER India’s tenth birthday, prompting us to celebrate its success but also look to the future. ASER in India has been ground-breaking, inspiring participatory learning assessments across the globe: Uwezo in Kenya, Tanzania, and Uganda, Bèekunko in Mali, Jàngandoo in Senegal, and ASER Pakistan. The findings of these assessments are widely cited and underpin important commentary on learning in the EFA Global Monitoring Report. Collectively, this movement has been critical in shifting attention away from the exclusive focus on access, brought about by the shape of MDG 2,  to one on learning embedded in the post 2015 sustainable development goals and the learning metrics task force.

But what of ASER’s impact on affecting reform efforts? A 2014 RCT impact assessment of ‘Uwezo’ in Kenya concluded the programme had no discernible impact on either private or collective action. This finding echoes a comprehensive survey of community led initiatives by The Global Partnership for Social Accountability, which warns that information alone is not enough to affect change.  ASER India’s 2006-14 review of learning trends also tells a disappointing story – at best of learning stagnation – giving rise to two questions – Why might that be?  What might be done about it?

On the ‘why’, my hunch would be that India’s disappointing ‘learning trajectory’ emanates from technical, socio-economic and political issues – including:

• Not long enough? Seeing positive change in national learning data in a country with 259 million school students will take time, especially when many new school entrants are first generation learners from the most disadvantaged communities.
• Not surprised? The ‘Uwezo’ study revealed parents were not surprised by findings – after eight year of ‘bad news’ some might argue that ASER India may have lost its shock value.
• Community not empowered and equipped? There’s a big difference between ‘knowing’ and ‘acting’ (as all dieters are aware). For progress you need ‘motivation’ and ‘mechanisms’ to make change happen.
• Government not engaged?  Like it or not, in most countries the government is the main provider of education. A universal truth for improvement anywhere is a community with ‘voice’ and a government with ‘ears’ to listen and ‘teeth’ to act.

So what might be done?

1. Move beyond bad news … bad news fatigue undermines motivation, displaces engagement with solutions, and can alienate. While media attention focuses on the ‘raw numbers’, much less is made of ASER’s broader policy relevant contributions spanning primary to secondary and teacher to out of school variables. 

2. Triangulate – ASER’s 2011 report cleverly referenced a range of learning assessments concluding that “broadly all indicate that learning is poor in Indian Schools”. This, approach made it harder for critics to dismiss ASER’s findings or allege that they were overly promoting offerings of its parent NGO, Pratham. There’s lots of compelling data out there, including the government’s own National Achievement Surveys. These broadly corroborate ASER’s findings. Further, they provide enriching insight on where learning challenges are by subject, and, are increasingly looking at the ‘spread of performance’ by gender, rural-urban and social group (there’s lots hiding underneath performance averages in India!). 

3. Seek government buy in:  In 2011, the ASER India report was launched by the then Minister of Education. This year, there was no high level ministerial engagement. ASER figures do feature in influential government publications, e.g. the 2015 Ministry of Finance Economic Survey and ASER/Pratham is a partner with some states. Securing government engagement isn’t always easy – but it is critically important.  Perhaps there is learning to be gained from ASER Pakistan and ‘Uwezo’ in E. Africa where government involvement appears more active. 

4. Strengthen Comparability: ASER is a monumental annual multi-lingual undertaking. Add to these changes in test forms and test protocols over the years and margins of error are inevitable. Advances in psychometrics mean there is always scope for improvement in comparing across time and geography. ASER has begun a technical dialogue on this issue with recommendations captured in the last chapters of a recent British Council Report.   For the future, ASER must move beyond sampling only rural schools as India is projected to be 50% urban by 2030.  

5. Go beyond benchmarking: Consider complementing ASER with smaller sample assessments which get granular about where the learning impediments lie. Build a virtuous cycle of assessment to learning action – and pay particular attention to strengthening the link between diagnosis and action.
6. Reach out – use ASER India to engage others: India’s learning challenge is massive, but in comparison to say environment, it is one around which most everyone is prepared to unite. New ‘crowd sourcing’ type approaches such as Ideo offer exciting ways to harness India’s drive and creativity. And there’s already plenty out there – from enlightened philanthropy such as Central Square Foundation through teacher led approaches such as STIR.As we shift to the new Sustainable Development agenda, we all need to recognise that delivering learning in classrooms is a far bigger challenge than building them. It is an exciting endeavour, with big prizes when we get it right, but as DFID’s 2013 learning framework shows it is a multi-dimensional challenge and complex challenges aren’t prone to quick fixes!India’s size makes for a massive learning challenge. And yet the country is blessed with an incredible capacity for innovation. Success lies with the energy, drive and creativity of its dedicated officials, citizens and NGOs.  There’s plenty of room for collaboration and ASER is well placed to play a catalytic role. Knowing the ASER team – they’re not ones to rest on their laurels – lots more to come in the next ten years!