20Jul2024

PAL Network Highlights 2023

Contacts

Le Mac Building |1st Floor Suite 1| Westlands

 vmiguda@palnetwork.org

+254 734 620 181

CITIZEN-LED ASSESSMENTS: A LOW-COST MODEL TO MEASURE FOUNDATIONAL LEARNING AT SCALE

WWHGE Technical Training Module by the PAL Network | March 2024

The What Works Hub on Global Education (WWHGE) provides a global platform to support governments who want to transform their education systems in a sustainable way, maximising the impact of education evidence to inform cost effective reforms to benefit all children. To support in the provision of this global platform, the People’s Action for Learning (PAL) Network has designed a technical training module on citizen-led assessments (CLAs), the signature assessment of the PAL Network, for the capacity building of governments, practitioners and stakeholders in the global education sector. This technical training module introduces and explains in detail what the CLA model is, and a guide on how to conduct CLAs.

This technical training module consists of four micro-modules, with each micro module being a video of ~20 minutes in length that explains (i) the history and relevance of CLAs, (ii) how to design and contextualise CLAs, (iii) how to conduct CLAs in practice, and (iv) introduce common assessments, a special form of CLAs. Each micro-module is supported with relevant documentation, including learning outcomes, further reading and resources, and a few questions to check understanding. All resources under this module are open-access under a CC-BY-4.0 license.

Watch module introduction and overview here.

Micro-Module Navigation:

MICRO-MODULE 1: WHAT ARE CITIZEN-LED ASSESSMENTS?. 2

MICRO-MODULE 2: HOW DO YOU DESIGN A CITIZEN-LED ASSESSMENT?. 4

MICRO-MODULE 3: HOW DO YOU CONDUCT A CITIZEN-LED ASSESSMENT?. 6

MICRO-MODULE 4: THE COMMON ASSESSMENT, A SPECIAL TYPE OF CITIZEN-LED ASSESSMENT. 8

MICRO-MODULE 1: WHAT ARE CITIZEN-LED ASSESSMENTS?

Speaker: Baela Raza Jamil, CEO, Idara-e-Taleem-o-Aagahi and ASER Pakistan

The Citizen-Led Assessment (CLA) model was pioneered in 2005 by Pratham, a prominent civil society organisation dedicated to enhancing educational outcomes in India. This innovative approach was born out of a desire to comprehensively understand and improve the educational landscape in India, where a paradox existed: many children were now enrolled in school, but there wasn’t any evidence that they were learning. In response to this challenge, Pratham embarked on developing a straightforward, yet highly effective assessment tool to gauge children’s reading and numeracy skills.

This approach was a fundamental departure from conventional, school-based assessment methods. Trained citizen volunteers took on the role of assessors, relying on oral assessments to avoid any assumptions about a child’s ability to read or write, and conducting assessments one-on-one with the child. The assessment’s location in the child’s household, rather than a school, was deliberate, ensuring a more inclusive representation of children in the sample and that assessments took place in safe environments. This approach transcended the boundaries of school enrolment, enabling the assessment of children not enrolled in school or with irregular attendance. This allowed for holistic mapping of the educational landscape rather than focusing solely on those who were part of formal education systems. Lastly, the citizen-led approach involved directly sharing insights on children’s capabilities with parents and community members, fostering accountability for education institutions at the grassroots and empowering community.

After completing the micro-module, you should:

  1. Understand the origin and purpose of CLAs.
  2. Recognise the difference between CLAs and other assessments.
  3. Learn the impact of CLAs in communities, schools, countries and worldwide.

See the recorded video here.

Ø  The slides used in this micro-module can be accessed here.

Further reading and resources:

  1. Introduction to and impact of CLAs in South Asia: Bhattacharjea, S., Saeed, S., Timalsina, R., & Ahmed, S. 2021. Citizen-led Assessments: A Model for Evidence-based Advocacy and Action to Improve Learning. Using assessment data in education policy and practice: Examples from the Asia-Pacific. https://unesdoc.unesco.org/ark:/48223/pf0000378149.locale=en.
  2. Motivation of CLA movement in East Africa: Twaweza East Africa. Part 1: Introduction. Uwezo Toolkit. https://www.youtube.com/watch?v=3HHBsGjSY_c
  3. Global spread of the CLA model: PAL Network. Spread and Use of Citizen-led Assessments. https://palnetwork.org/wpfd_file/2020-pal-spreadanduseofcitizen-ledassessments-summary/
  4. Impact of ASER, the pioneering CLA: Chaudhry, R. 2023. Shifting from Schooling to Learning: How ASER Shaped India’s Education Discourse and Policy by Measuring Learning. RISE Programme Blog Series. https://riseprogramme.org/blog/shifting-schooling-learning-aser-shaped-indias-education-discourse-policy-measuring-learning.html
  5. Communicating evidence from CLAs for impact: Twaweza East Africa. Part 6: Communicating Evidence. Uwezo Toolkit. https://www.youtube.com/watch?v=LZoZLbPn5ao

Questions to check your understanding:

  1. What are the key principles of the CLA movement?
  2. How does the CLA model differ from conventional, school-based assessments?
  3. What is the theory of change of the CLA model?

MICRO-MODULE 2: HOW DO YOU DESIGN A CITIZEN-LED ASSESSMENT?

Speaker: Prof. Felipe Hevia, Director, Medición Independiente de Aprendizajes (MIA)

The design of citizen-led assessment encompasses various components, including test development, sampling, training, survey conduct, and quality controls. The assessment tools aim to understand a child’s abilities and mastered skills, classifying them as beginners, letters, words, paragraphs (grade 1 level), and stories (grade 2 level) in reading. Similarly, the math assessment categorize children based on number recognition (one or two digit) and operation levels. The assessments are administered orally and individually and require about 10 minutes per child. To develop the assessment tools, education and curricula experts are engaged to design the assessment items based on the country’s curriculum. To sample for citizen-led assessments, PAL Network members sample 30 villages in a district, randomly selecting 20 households in each village, resulting in 600 sampled households per rural district and about 300,000 households nationally.

After completing the micro-module, you should:

  1. Recognise the steps for adapting CLAs to a new context.
  2. Understand the sampling approach used for CLAs.
  3. Learn the training procedures for CLAs and which resources are provided to surveyors.

See the recorded video here.

Ø  The slides used in this micro-module can be accessed here.

Further reading and resources:

  1. Comprehensive manual on implementing CLA model: PAL Network. 2017. PAL Network Starter Toolkit: Planning and Executing a Citizen-Led Assessment. https://palnetwork.org/wp-content/uploads/2024/03/PAL-Network_Starter-Toolkit_Planning-and-Executing-a-CLA.pdf
  2. CLA resources across all PAL Network countries, including datasets, instruction manuals and open-access tools in local languages [webpage]: PAL Network. Citizen-Led Assessments. https://palnetwork.org/citizen-led-assessments/
  3. Selection of partners and volunteers: Twaweza East Africa. Part 2: Partners and Volunteers. Uwezo Toolkit. https://www.youtube.com/watch?v=PqaHVxyXYw8
  4. Sampling
    1. South Asia using ‘village mapping’ method: Wilima Wadhwa. 2022. ASER Sample Design. https://asercentre.org/wp-content/uploads/2022/12/sample-design_2022.pdf
    2. East Africa using ‘houselisting’ method: Twaweza East Africa. Part 3: Sampling. Uwezo Toolkit. https://www.youtube.com/watch?v=UEPEBp4Qypc

Questions to check your understanding:

  1. What are the key components to consider when adapting the citizen-led assessment model to a new context?
  2. What are the steps to adapting the assessment tool to a new context?
  3. What is the sampling approach used in CLAs to ensure representative data?

 

MICRO-MODULE 3: HOW DO YOU CONDUCT A CITIZEN-LED ASSESSMENT?

Speaker: Faridah Nassereka, Senior Program Officer Assessment, Action, and Research, Uwezo Uganda

The design of citizen-led assessment encompasses various components, including test development, sampling, training, survey conduct, and quality controls. During the conduct of the survey, volunteers visit villages in pairs to engage directly with the community and parents to obtain consent for the assessment and administer the assessment to the child accurately and as trained. Data collection involves village mapping school visits, enumeration-level data gathering, and assessments with monitoring, focusing on areas with weaker volunteers. Continuous communication with the partners, coordinators and volunteers is essential to resolve any issues that might arise during the assessment process. Post-assessment activities include monitoring and rechecking, implementing Lot Quality Assurance Sampling (LQAS), and developing criteria for including or excluding villages in the final data set. Reflective meetings and feedback sessions should be done to get partner and volunteer feedback.

After completing the micro-module, you should:

  1. Learn the steps involved in preparing for a CLA survey.
  2. Understand what is needed during survey conduct.
  3. Internalise the importance of quality control and how this is promoted in the CLA process.

See the recorded video here.

Ø  The slides used in this micro-module can be accessed here.

Further reading and resources:

  1. Instructions for conducting a CLA in the field: ASER Centre. Do It Yourself ASER. https://asercentre.org/wp-content/uploads/2022/12/Do-It-Yourself-ASER-DIYA-English-Final_2023.pdf
  2. Process of testing and grading a child: Twaweza East Africa. Part 4: Testing and Grading a Child. Uwezo Toolkit. https://www.youtube.com/watch?v=22ELrgQlcC4
  3. Process for ensuring data quality: Twaweza East Africa. Part 5: Ensuring Data Quality. Uwezo Toolkit. https://www.youtube.com/watch?v=bgl7c_P3LiU
  4. Data quality standards governing CLAs and other PAL Network assessments: PAL Network. 2018. Data Quality Standards Framework. https://palnetwork.org/wpfd_file/2018-pal-network-dqsf-final/

Questions to check your understanding:

  1. What are the steps involved in preparing for conducting a CLA survey?
  2. What role do volunteers play durng the survey process?
  3. What post-assessment activities are conducted to ensure high data quality and analysis in the CLA model?

MICRO-MODULE 4: THE COMMON ASSESSMENT, A SPECIAL TYPE OF CITIZEN-LED ASSESSMENT

Speaker: Rastee Chaudhry, Assessment Manager, PAL Network

The development of common global educational goals and the need for comparable data to monitor education quality targets have meant that many low- and middle-income countries face increasing pressure to participate in existing international and regional assessment programs. Existing international and regional assessments are based on models and methods that emerged in the context of Global North countries, which have characteristics that are often very different from those of Global South countries.

Since 2018, the PAL Network began designing ‘common assessments’ – a special type of CLA, to respond to the global need. ‘Common assessment’ refers to the same assessment format and items being used across country contexts to promote rigorous cross-country comparisons and learning. The International Common Assessment of Numeracy (ICAN) was the first proof of concept of common assessments and was administered in 2019 and 2022 in 13 and 7 countries respectively. Currently, the PAL Network is finalizing the design of the Early Language & Literacy and Numeracy Assessment (PAL-ELANA), a digitally adaptive assessment of numeracy, language, and literacy, and will be scaling it to 12 countries in 2024. The Network has also initiated work on ICAN and ICARe, a large-scale common assessment of numeracy, language, and literacy in 15 countries to provide nationally representative estimates of foundational learning to check progress against SDG 4 targets.

After completing the micro-module, you should:

  1. Recognise the varying demands from data and why different assessment types are needed.
  2. Appreciate the difference between common assessments and traditional CLAs.
  3. Learn about the PAL Network’s common assessment portfolio.

See the recorded video here.

Ø  The slides used in this micro-module can be accessed here.

Further reading and resources:

  1. Comprehensive overview of PAL Network assessments from CLAs to common assessments: PAL Network. 2023. Assessment for Advocacy: How decades of evidence from the PAL Network has been transforming communities, programmes, and policies in the Global South. GEM Report Spotlight Series. [Forthcoming]
  2. All resources on ICAN [webpage]: PAL Network. International Common Assessment of Numeracy (ICAN). https://palnetwork.org/ican/
  3. All resources on PAL-ELANA [webpage]: PAL Network. Early Language & Literacy and Numeracy Assessment (PAL-ELANA). https://palnetwork.org/elana/
  4. All resources on ICAN 2.0 and ICARe: PAL Network. ICAN and ICARe. https://palnetwork.org/ICAN-and-ICARe/

Questions to check your understanding:

  1. How does the PAL Network’s common assessments address the varying demands for data in education?
  2. How are common assessments different from the traditional citizen-led assessment model?
  3. How do the PAL common assessments contribute in the global efforts of monitoring progress towards SDG 4?