EnglishKenyaPAL Network
In a previous blog, we argued that the market for learning assessment is very inefficient and therefore warrants public action. As things stand:
- More than half of countries do not participate in a cross-national assessment, makes it hard for them and for the international community to benchmark their progress towards the learning outcome indicators in SDG 4.
- Countries that may want to participate in a cross-national assessment, and agencies that could cover the cost, both face obstacles standing in the way of effective and cost efficient solutions.
Today, we want to contribute further by proposing a series of possible solutions for five forms of inefficiency and the problem of inequity. While the solutions have different political and monetary costs, they are all relatively easy to adopt. And, in an ideal world they would all be carried out more or less simultaneously as they are all highly complementary with each other.
Inefficiency 1: Countries do not have all the information to make an informed choice over the assessment product they need
Solution: Produce a consumer guide on the different types of assessments. The current lack of information means that countries must do their own digging to evaluate the options. At a minimum, they need to understand how international assessments can be used to: support the national agenda; report on the SDGs; and help their ministries make management decisions.
Ideally, countries want an assessment to meet all these needs in one go. But finding the right fit can be difficult, which is why they need better guidance. For example, a country might be thinking about participating in an international assessment even though it has a national assessment at the same education level. To evaluate the options, it needs to compare the overall costs and benefits as well as more technical information to decide on the nitty gritty questions such as items and constructs. As with any expensive product, countries need a consumer guide, providing accurate and objective information, to compare options.
Inefficiency 2: The cost of participation in cross-national assessments is neither transparent nor uniform
Solution: Provide more transparent price information. In the current market, consumers often have to haggle, like at a roadside vegetable stall, rather than choosing their products at a market with prices clearly posted. While the haggling can lead to better prices for some, there tend to be hidden expenses and higher transactions costs, and maybe higher prices for others. We are all for negotiations, but they should be based on transparent pricing information. An official body could produce accurate, comprehensive, and up-to-date information on the current costs of assessments, while documenting steps or conditions that can help countries negotiate.
Pricing information should at a minimum include: fees paid to the international assessment organization; opportunity costs of the technical knowledge required within national agencies; financial costs of field deployment; opportunity costs of field deployment if existing staff are used. Today, there is no clarity on how much it costs to develop an assessment item, for example, or to conduct a pilot test. And some of the information is actually contradictory. For example, countries are often led to believe that by joining an international assessment they will benefit from economies of scale. Yet why is it that the fees never seem to go down as the pool of participants grows?
Inefficiency 3: Countries don’t necessarily have the tools use their assessments for global reporting
Solution: Develop methodologies to link assessments to a global minimum proficiency level. If countries and development agencies know more about how one assessment “translates” into another, they would not feel the same pressure to “buy” every possible test, in addition to their own national assessment. Countries can better evaluate the options and their relevance to policymaking if they can see the level at which an assessment is linked to a global yardstick. If one knows how to translate metres into yards, one does not need two measuring sticks. This information will also help to boost the technical skills of national staff while also supporting development partners that fund assessments as part of their contribution to SDG 4.
Inefficiency 4: Technical barriers prevent producers (including countries themselves) from entering the market
Solution: Create item banks to help build sensible assessments that can be used for international reporting.
One would expect that with 25 years of intensive experience of preparing cross-national assessments, it would be straightforward to copy this process. Nobody argues that absolutely all information should be in the public domain, as it is important to ensure assessments are valid and keep certain aspects confidential. But the process of producing assessments, and the background knowledge needed, has been sufficiently standardized that the current high costs are not justified.
One step towards generating capacity in the countries and lowering costs is the use of artificial intelligence applied to a bank of items properly aligned to the different education cycles of countries. The platform would embed a bank of questions with known and tested technical properties. The approach would also suggest ways to combine items in a sound manner, as needed for reporting. Countries would be able to get and contribute items, to build the whole assessment if needed, to run the modeling that produces the results without having to commit themselves to packages where their ownership and voice is reduced. A machine learning engine would, in an ambitious scenario, allow adaptive testing at low cost. Similarly, one way forward would be to develop a pool of items from existing assessments, as a free global public good that would empower national authorities to develop their own assessments.
Think of it in terms of a generic drug that is only slightly different to a costly branded alternative. The item bank and associated software tools could help new providers, or countries who would self-provide, in a manner similar to how the global community has helped generic drug manufacturers develop products.
Inefficiency 5: The processes whereby consumers and producers interact is a black box and cannot be replicated
Solution: Create more physical marketplaces. Most of us like touching and feeling things we buy. If we are buying a bicycle or car, it is sensible to try it – even if we end up making the final purchase online. The learning assessment market should offer the same experience – a place where users, producers, and international organizations can meet and make sales pitches. Some of this already happens at conferences, with side events organized by producers that try to educate consumers (but few where consumers try to educate producers, itself a cause of market failure). But perhaps there ought to be more physical market spaces to allow not only comparison shopping, but also including exchanges where consumers can learn more about how assessments are produced and why the good ones might not need to be as expensive as one might think.
Inequity: Available resources to participate in cross-national assessments and develop national capacities are allocated unequally.
Solution: Transfer purchasing power and technical skills to countries. As noted in the previous blog, even the most efficient markets don’t care about whether consumers enjoy a product according to their needs. Assessment costs are high, relative to other discretionary quality assurance mechanisms, for poorer countries. They may need subsidies either to cover the cost of both participation and technical skills development, or as a behavioural nudge. As the price of cross-national assessments becomes standardized and transparent, international agencies should ensure grants are made available to countries with a long-term perspective, to ensure all countries benefit equally according to their need. The design of such a scheme is non-trivial and needs to be thought about carefully so as not to create further inequalities or perverse incentives (e.g., pretend to a lack of interest so as to merit the behavioural nudge).
We hope that these two blogs have provided some options for improving the market for assessments. We have tried to be measured and non-controversial. Yet at the same time, we want to spur some debate. So tell us what you think by posting comments on this blog and joining the conversation at #LearningMarket.
[1] Full disclosure: the second author is associated with the origins of EGRA and EGMA.