A Year in Review: Ensuring quality data across the PAL Network

The citizen-led assessment model was born in India in 2005 when India’s largest NGO – Pratham, designed an innovative new approach to assess the basic reading and numeracy competencies of all children, regardless of their schooling status. Over the past eleven years, the citizen-led assessment model has gained momentum across the global South, where it has been borrowed and adapted to country contexts where not all children are in school and learning.

The citizen-led assessment model was designed carefully and deliberately to cater to the realities found in global South countries. Using trained citizen volunteers, simple tests and tools and assessing all children are all fundamental aspects of the citizen-led assessment model. The design reflects a philosophy that is different from that of standard school-based assessments that have been developed in western country contexts.

The apparent simplicity of the citizen-led assessment, which enables large numbers of ‘ordinary citizens’ to get involved, is backed up by sophisticated ‘behind the scenes’ monitoring processes designed to ensure that the data generated are reliable. This includes systematic processes for sampling, partner and volunteer selection, training, monitoring and recheck. In addition, careful data cleaning and other methods are used to validate the data. However, due to the organic expansion of the network over time, these processes are not consistent across PAL Network countries, nor visible to people outside the network.

January 2017
January marked the beginning of a newly developed three-year strategy for the network. Focused on ensuring the collection of high quality data, country leaders and data analysts convened in Aurangabad, India for the network’s first workshop on data quality. At the workshop, participants agreed on the need to produce a data quality standards framework. As a starting point, the PAL Network Data and Design Working Group drafted a set of ‘minimum standards’. In addition, 62 documents pertaining to data quality were collected from the network members to inform a larger Data Quality Standards Framework (DQSF).

First Data Workshop in Aurangabad, India

March 2017
The draft minimum standards document was tabled at the 4th PAL Network Steering Committee meeting in Mexico City and further explored during the Data and Design Working Group meeting in Xalapa, Mexico. Participants mutually agreed on four next steps:

1. To develop a ‘rapid analysis’ framework to analyse the 62 existing documents pertaining to data quality
2. To conduct a landscape review of existing international and regional assessment data quality standards
3. To draft the first DQSF combining the research and analysis outlined in steps 1 & 2
4. To reconvene the appropriate experts to finalize the DQSF and draft an implementation strategy

Working Group Meeting in Xalapa, Mexico

May 2017
The PAL Network Secretariat undertook the initial analysis of the existing network documents, with follow-up Skype interviews conducted with data analysts across the network.

July 2017
The National Foundation for Education Research (NFER) was identified to conduct a landscape review of existing international and regional assessment data quality standards to inform the creation of the DQSF. The PAL Network Data and Design Working Group and the Network Secretariat then worked in consultation with NFER to draft the DQSF.

November – December 2017
Consultation between Data and Design Working Group and Secretariat team began in preparation to hold the second data workshop in 2018. The workshop will once again convene country leaders and data analysts to finalize the DQSF document and begin planning for the implementation of the DQSF, starting with self-reporting and peer-monitoring mechanisms.

Looking forward: Building on the foundations laid in 2017
The Data Quality Standards Framework (DQSF) is a central element to the network-wide commitment to producing high quality, robust data to be reported at regional, national and international levels. The DQSF will help PAL Network countries to improve technical rigour, whilst allowing flexibility to accommodate the diversity of processes and adaptations to local context that is central to the citizen-led assessment model. The creation of the DQSF will be accompanied by implementation and monitoring plans, where member countries will support each other to meet the minimum required standards, learn from best practice across the network and help target technical support. The DQSF aims to be clear and accessible to all users, no matter how familiar they are with citizen-led assessments.