The learning assessment market: pointers for countries – part 1

By Silvia Montoya, Director, UNESCO Institute for Statistics (UIS) and Luis Crouch, Senior Economist, RTI International [1]

Measuring learning outcomes is key to the Sustainable Development Goal for education (SDG 4). There are about a dozen indicators that measure learning outcomes.

Data for these indicators are provided via a market. It may seem odd to think so but think about it for a moment: there are data producers, there are data consumers (countries, policymakers, international agencies and researchers), and there are goods and services exchanged for money (prices) to produce the assessment data.

Wikipedia, for example, characterizes markets as institutions that “facilitate trade and enable the distribution and resource allocation in a society. Markets allow any trade-able item to be evaluated and priced. A market emerges more or less spontaneously or may be constructed deliberately by human interaction in order to enable the exchange … of services and goods.”

While the specifics of a market will obviously vary, there are two central questions: does it allocate resources efficiently and equitably? In this blog, we ask this of the learning assessment market, and find the answers fall short.

Is the market for learning assessments efficient or equitable?

Let’s start by efficiency. In an efficient market:

1. There are many producers and consumers competing with each other over the same product
2. The products traded are uniform and undifferentiated
3. Consumers have all the information they need about the products including details on how they are produced (in economists’ jargon, there is no ‘asymmetry of information’)
4. The production techniques are known to all and can be copied
5. There are no significant economies of scale or barriers to entry
6. There is no price discrimination: prices are transparent and uniform

But with learning assessments, there is product differentiation. In fact, no important ‘product’ sold in the learning assessment market is the same as any other, and organizations purposefully differentiate. Some assessments are about skills needed for the labour market, others are curriculum-based. Some are designed for primary education, others focus on lower secondary. Some are citizen-led, others are government-led. And so on.

Consumers are not well informed about how different learning measurement products are made and how they correspond to their specific needs. They have a hard time evaluating the different prices and efforts required to develop a reasonable product. Producers cannot easily copy each other, and the knowledge needed to produce a good quality product is arcane.

There are also significant barriers for possible competitors to enter the market because it is costly to build a set of good learning assessment questions. New providers typically emerge only to provide a differentiated product. For example, there are assessments serving different geographies (such as initiatives in East Asia) or offering different ways of administering and engaging with the community (citizen-led assessments) as well as different education levels (e.g. the Collegiate Learning Assessment, a higher education standardized test in the United States).

There is also price discrimination. Not all countries pay the same. There is some negotiation on price and different levels of subsidies. There is also intermediation. Prices in many cases are negotiated between third party payers (e.g. development partners) and the producer. This can be a good thing in some ways (e.g. the poor pay less) but it also results in non-transparency of prices.

Now let’s turn to equity. The bottom line is that markets don’t care about equality or justice. A market may simply not work for those without purchasing power. A market can have a major equity problem and still not be considered a failure in the classic sense.

No real world market fulfils all of the conditions listed in the Wikipedia article.  In most cases, some of the conditions cited above are violated at least a little.

The problem is that all the conditions are violated in the market for learning assessments—and not just by a little bit: at least some characteristics are the diametrical opposite of those needed for efficient and equitable markets.

How to fix a market failure

When markets are inefficient in these ways, it is said that there is a “market failure.” This could justify public or collective action.

Someone could argue for a public takeover of these functions. Another approach is “market-shaping”: the public sector can induce markets to be more efficient. In the health sector, international organizations shaped the market of anti-retroviral drugs for HIV/AIDS to improve the production, procurement, and prices. This has been, arguably, among the most successful single interventions of the past 30 years or so probably helping save millions of lives.

To improve both efficiency and equity in the assessment market, think of three solutions:

One is to systematize market information by publicizing more information for users about which assessment is fit for purpose, what is a reasonable price to pay etc. Various public agencies, such as the World Bank, have published such guides. More could be done, especially with regard to price information, and more could be done to disseminate this knowledge.

Another idea is to create free information on how to craft a decent assessment. An item bank of assessment questions that work well and information on how to put such questions together might be an example. A similar idea is to lay out technical policies and procedures for assessments, such as for citizen-led assessments or early-grade assessments. The UIS has started working in this direction but more could be done.

Finally, from the equity point of view, it would be useful to consider transferring purchasing power to those who need it, either because they are very poor or simply because they need the behavioural nudge. But such subsidy mechanisms must be transparent, explicit and designed carefully.

This is an important discussion for anyone working on SDG 4. The global desire to make progress on learning, and not just access, is a central pillar but may also have, perhaps unwillingly, contributed to the market we find ourselves with today. But markets don’t care about justice or equality; they are rarely set up to work for those without purchasing power. Something must be done, therefore, to level the playing field.

Stay tuned for more details and ideas.

Meanwhile, we welcome your feedback here and using #LearningMarket.

[1] Full disclosure: one of the authors is associated with the origins of the assessments EGRA and EGMA.

Blog originally posted on the World Education Blog