A Common Assessment to Measure Foundational Numeracy: How NewGlobe’s Leverages ICAN to Monitor Learning on a Global Scale

By: NewGlobe (Daniel Rodriguez-Segura and Melanie Gaudet) and PAL Network (Rastee Chaudhry and Najme Kishani)

Reliable and actionable data on learning outcomes is pivotal for designing learning-responsive policies and practices that improve educational achievements, especially in low- and middle-income countries. This is particularly important, as available data shows that the learning crisis persists; for example, over half of children (56%) completing primary school have not yet mastered the minimum foundational numeracy skills expected at the primary level (UNESCO Institute of Statistics, 2019).

Tracking foundational learning globally, especially at the lower primary level, faces two major challenges. First, data on learning outcomes is relatively scarce in general, as many countries around the world lack assessments that track this (for example, Crawfurd, Le, & Sandefeur, 2023). Second, even the relatively scarce data that is collected is often gathered in a way that is not comparable across countries or even within the same countries over time—usually due to shortcomings in the assessments used or the scale at which they were administered. These challenges restrict the ability of policymakers to make informed decisions to confront the learning crisis in an evidence-driven and real-time manner. Therefore, expanding the quality and quantity of available data is essential for governments and organisations seeking to improve learning outcomes.

The International Common Assessment of Numeracy (ICAN) is a viable alternative for policymakers and organisations keen on collecting quality data on foundational numeracy outcomes. ICAN is actionable, comparable across time and regions, aligned with international frameworks, and yet, it is manageable to administer at scale. This blog describes ICAN, and how a global organisation — NewGlobe — has successfully integrated ICAN into its cycles of evidence creation to diagnose and measure learning outcomes at scale.

ICAN: A practical assessment tool that merges versatility and robustness

The PAL Network designed its first common assessment tool ICAN (the International Common Assessment of Numeracy) in 2018, which builds on PAL Network’s signature Citizen-Led Assessment model. ICAN is an easy-to-use and scalable tool that measures a range of foundational numeracy skills in accordance with UNESCO’s Global Proficiency Framework,  pegging it to an established international framework. ICAN builds upon preceding assessment methodologies by enhancing the robustness of its item bank and establishing item commonality across nations, administration practices, and analytical procedures. Consequently, this facilitates comprehensive comparative analyses, fostering the advancement of learning-focused practices within various educational contexts.

ICAN is a public good open for use across contexts and organisations. In recent years, ICAN has been used extensively by NewGlobe, an organisation that partners with governments in Africa and Asia as a technical advisor to holistically transform education systems using a comprehensive reform model. NewGlobe has adopted ICAN since 2020 to generate insights into the education systems where it runs its multi-year holistic education reform programmes.

This blog covers how NewGlobe leverages ICAN to quantify learning outcomes at scale and over time in a robust manner, and as such, serves as a case study of how PAL Network’s common assessments can support governments and civil society in monitoring learning outcomes soundly and consistently.

Evaluation of best practices at NewGlobe: A case study of using ICAN to create adaptable, learning-centric policies

NewGlobe’s approach relies on a comprehensive reform model, encompassing structured pedagogy, teacher training, coaching, data-driven school management, and agile monitoring of learning outcomes across diverse contexts. Therefore, obtaining reliable, comparable, and actionable data on learning outcomes is crucial for the continuous delivery and improvement of NewGlobe’s programmes.

The use of ICAN to assess foundational numeracy outcomes in the contexts where NewGlobe works was informed by the delicate balance achieved by the ICAN tool between strong psychometric properties and the ability to communicate emerging outcomes in a clear and tangible manner to government partners and other stakeholders. ICAN’s alignment with the international benchmarks set by the Global Proficiency Framework also facilitates comprehensive cross-national comparisons through its open-source datasets from across the globe. This is especially pertinent to a suite of essential concepts for primary grades, the focus of the majority of NewGlobe’s programmes. Additionally, ICAN provides an extensive examination of numeracy subskills, yet it remains less time-consuming compared to more detailed but prolonged evaluations.

Here, we provide three concrete examples of how NewGlobe uses ICAN at scale:

Initial learning benchmarks. NewGlobe conducts an “Academic Field Study” prior to the launch of new partnerships. The purpose is to assess a random sample of schools and students representative of the entire region where the new programme will be launched. These studies assess foundational literacy and numeracy skills. To assess numeracy, NewGlobe uses ICAN and supports it with a lengthier, internally developed assessment to map students’ skill levels onto pedagogical levels in NewGlobe’s continuum of numeracy content. This is a necessary step in providing targeted support under the NewGlobe model. While the internal assessment covers a wider range of skills, ICAN offers a far more succinct and easy-to-understand diagnosis of learning challenges. For instance, Figure 1 shows an example of actual data collected by NewGlobe, comparing and contextualising the performance on a given subskill (simple addition of 32+15) across nine settings. In places where deeper remediation is needed, the depth of student learning deficiencies are best illustrated and communicated through ICAN, while the design of the remediation programme is informed by the detailed internal numeracy tool.

Consistently during these studies, there is a notable correlation between ICAN results and other learning measures, including reading fluency and comprehension, and the lengthier internal numeracy tool.This demonstrates ICAN’s ability to effectively identify regional learning trends and pinpoint areas requiring educational support more swiftly and straightforwardly than longer assessments.

Impact measurement. NewGlobe relies on ICAN for internal impact evaluations of programmes to measure and communicate the programme’s effectiveness to stakeholders, as well as to identify areas for further improvement. For this purpose, ICAN is almost exclusively used to measure progress in foundational numeracy. Typically, ICAN is administered to a representative sample of students at treatment and comparison schools at baseline and endline, with comparisons drawn between the growth trajectories in the two groups. Results have shown that ICAN produces quality data, with high reliability statistics across all school groups and data collection rounds. As expected, numeracy scores tend to increase over time for the same cohort of students in the same schools, although the rate of growth is significantly faster in schools supported by NewGlobe relative to the comparison group. Figure 2 compares the same set of schools longitudinally during two time periods.

Similarly, across all locations where ICAN has been administered in primary grades, true ceiling effects have not been encountered even up to Grade 6 (e.g., groups of children with numeracy skills above those being assessed in ICAN), despite considerable variation in the schools’ starting levels (e.g., Figure 1) and the size of the program’s effect in those locations. While this unfortunately speaks to the dire initial numeracy levels in many of these contexts, it also demonstrates ICAN’s appropriateness for assessing competencies at the most foundational level. NewGlobe has also found that disseminating ICAN results to government partners is remarkably straightforward and tangible, given the assessment’s foundation in commonly known mathematical subskills (e.g., “solving a word problem involving subtraction”). Moreover, the availability of data on ICAN outcomes in other contexts enables NewGlobe to meaningfully interpret the learning gains achieved in NewGlobe-supported schools by benchmarking the observed learning levels against performance in those other contexts. For instance, Figure 2 illustrates how a program has been able to advance children’s numeracy proficiency in Kwara state in Nigeria, relative to other comparable education systems.

In-depth research and programmatic decision-making. By now, NewGlobe has administered ICAN in 25+ rounds of data collection assessing 35,000+ pupils in numerous contexts, which enables the organisation to conduct deeper analyses that meaningfully inform its work in the longer term. For instance, NewGlobe recently conducted research on between-school heterogeneity of learning outcomes and the implications of this heterogeneity for curriculum-setting at a large scale (Rodriguez-Segura and Tierney, 2024). This research was enabled by the fact that ICAN was used consistently across all six school sites and four countries of interest for this study, which allowed NewGlobe researchers to make meaningful comparisons across locations that speak to the relative pedagogical needs of each place. As a result, NewGlobe found that between-school heterogeneity is indeed a significant pedagogical challenge in certain education systems, while not in others. These results reinforced the need for a dynamic programmatic and pedagogical solution to enhance between-school heterogeneity,  responsive to the varying learning levels in a given context. Overall, this work underscores the criticality of actively gathering timely, context-specific data to shape policies addressing between-school heterogeneity. The preliminary analysis illustrates the context-dependent nature of educational challenges and the tailored solutions they necessitate — which were vividly brought to life by the use of ICAN.

Call to action: With the right tool, a better measurement protocol is achievable and in-reach

The PAL Network and NewGlobe jointly believe that a renewed focus on more and better data is both necessary and achievable. This approach should be an essential pillar of the strategic plans of governments and organisations to facilitate stronger learning outcomes. ICAN is an accessible and effective resource that data-driven governments and organisations can use to fill the existing gaps in knowledge of learning outcomes and also to track progress towards national and international commitments, such as SDG 4.1.1a; it is only with this visibility that learning outcomes can  be improved.

ICAN is a very high quality tool for collecting data on foundational numeracy.  Especially due to it being public access, this tool removes barriers to large-scale learning insights and collecting quality, psychometrically sound learning data, and enabling stakeholders to reach their goals of catalysing learning gains while sharing data for continued support and inclusion.

Currently, the PAL Network is rolling out its next generation of common assessments: the PAL-ELANA (Early Language, Literacy, and Numeracy Assessment) that is a tablet-based multiple stage adaptive assessment of both language, literacy and numeracy for 4 to 10 year olds tested in 12 countries and available in 9 languages, and ICAN 2.0 and ICARe (International Common Assessment of Reading), which are paper-based updated versions of the original ICAN and incorporate language and literacy measures in 12 languages for use in 15 countries. These tools can be made available on request, and are well suited for use at the systems level or even school levels to support learning – the core mission of the PAL Network. For more information, please contact info@palnetwork.org