This week, Results for Development’s education team published a multi-year effort assessing the effectiveness and impact of citizen-led education assessments of student learning. The various initiatives and campaigns (“CLAs” for short) are part of a vanguard effort in the education and transparency communities to leverage volunteer-led testing of student competencies towards advocacy and educational policy reform. As we describe in the study, the core theory of change being brought to bear by these efforts is that by generating new data around (the lack of) student learning, the public will increase its demands for educational reforms, and policy makers and service providers will be compelled to respond to those demands through increased investments in and smarter design around key educational programs.
To summarize the key findings, we learned that:
- It’s indeed possible to carry out citizen- and volunteer-led assessments of student learning (at least to assess basic reading and math skills).
- The groups that carried out this work did it quite competently and with extremely limited resources.
- In certain contexts, particularly in India and Tanzania, these efforts contributed to a real shift in the national debate around learning and education, particularly around the need to focus efforts on educational outcomes (whether students learn) rather than solely on educational outputs (whether students attend school).
- In virtually all cases, there was no discernable impact of these efforts on student learning at the individual and local level. In other words, kids are learning at the same (highly deficient) levels in these places despite sophisticated CLA campaigns having taken place.
That last point offers some important lessons for transparency, accountability, and citizen participation (TAP) practitioners. While in some ways the key takeaways seem commonsensical, we seem to subconsciously re-engineer these same deficiencies time and again as a community.
“Build it and they will come” is a really poor strategy for achieving social change. Whether your “it” is educational data around student learning, open government data, or transparent budget information, simply “opening up” information and expecting parents, teachers, students, or the average “citizen” to pick it up and suddenly transform themselves into a sophisticated public advocate and policy wonk is a pipe dream. This suggests other simple but important design implications for future work in both TAP and CLA, including…
Infomediaries are extremely crucial for translating raw data into digestible bites that can motivate large-scale public mobilization around an issue, and also for gaining access to policy makers and politicians with the ability to change service delivery programs. Thus, working more closely with media outlets, expert non-governmental organizations, local think tanks, labor unions, and religious organizations becomes increasingly paramount in the future if we want to see the results of CLA efforts translate into policy and programmatic changes that stand a chance of positively affecting student learning.
Those carrying out CLAs might or might not be ideally suited to serving in that crucial infomediary role. Rather than automatically assume that overstretched CLA organizations are necessarily the right actors to translate CLA data into policy asks and programmatic recommendations, exploring potential partnerships with other allies – whether from academia, think tanks, teachers unions, opposition political parties, or religious organizations – seems worthy of experimentation.
All of these (again admittedly straightforward) insights offer lessons for the broader open data and transparency communities of practice. How many open data portals have been naively built with the assumption that “they” (whether “citizens” or software developers) would “come” and enthusiastically turn raw data into social change? Why are TAP practitioners still struggling to build strategic relationships with media houses and other thought leaders as a way to amplify their messages and de-wonkify their work? And why do we often assume that TAP-centric organizations themselves are the ones best situated or with the right credibility to make the crucial asks of policy makers and political leaders; why can’t we construct more effective broad-based alliances and coalitions for reform?
These aren’t simple questions to answer in any context, of course. But with this new set of analyses of CLAs, we have a fresh set of inputs with which to sharpen our thinking.