Deconstructing PISA: Implications for Education Reform and Fighting Poverty

This post first appeared on The Huffington Post.

Every three years, American policymakers eagerly anticipate the release of scores for the Programme for International Student Assessment (PISA). While any single test, no matter how strong, can explain only a limited amount about our education system, PISA provides some unique insights, testing students’ ability to apply knowledge and skills both in and out of school. It is taken only by 15-year-olds, making it a decent proxy for the “college-and-career readiness” that is the focus of current debates.

The 2013 headline is basically that the United States falls right in the middle of the pack, as it has for several decades. The US Department of Education and its allies used those rankings to argue that US schools are “stagnating” and to advance specific reforms it says will fix them. However, average scores may obscure and confuse more than they inform. Indeed, scores from individual states that have their own PISA rankings offer more policy-relevant insight than overall US rankings. This makes sense — US education looks more like a diverse patchwork than a unified system. Public investments in schools, and in students and their families, vary greatly across states, as do other policies that may boost or depress scores. Luckily, this year, three states received individual PISA rankings — as if they were independent countries. This can help us connect the dots between those disparities and scores.

Massachusetts is the good news story. If it were its own country, it would rank sixth in reading of 65 countries and economies included, behind only Singapore, Japan, Korea and the Chinese regions of Shanghai and Hong Kong. Its students rank just above Finland and Canada, some of the world’s best readers. Though its math scores are slightly lower, Massachusetts keeps company with Belgium and Germany and is only slightly behind Finland and Canada, ranking 16 of 65. In science, Massachusetts ranks 11th, ahead of Canada and Germany. Connecticut, the second of three states with its own scores, falls just below Massachusetts, ranking 9th in reading, 18th in math and 17th in science.

On the other hand, Florida’s students rank below the US average in all three categories. They are 27th in reading, ahead of Austria, but behind the Czech Republic. In science, Florida is 38th, behind Spain, Portugal and Italy, and just ahead of Slovenia. Florida’s math scores are the most troubling. If it were a country, it would rank 41st of 65, behind Croatia and ten points lower than Hungary.

How might these more nuanced findings inform policy? The Department of Education advocates raising academic standards, using student test scores to evaluate teachers, and implementing school “turnaround” policies that include firing staff, closing schools and converting them to charters. More recently, it has made quality early childhood education a high priority. If Massachusetts is a good indicator of what works, high standards help. However, attaining the benefits of high standards is contingent on first laying a foundation of comprehensive student, family and school supports. The same is true of turning around schools — years of investment in access to quality pre-kindergarten programs, physical and mental health care and extra support for teachers serving in struggling schools appear to be key ingredients in the state’s recipe for success.

Florida looks very different. Not only teachers, but entire schools are evaluated using test scores. Indeed, the state recruited Tony Bennett to bring his Indiana-based concept of giving schools A-to-F grades. It has the seventh-highest percentage of students in charter schools, nearly six percent, versus a national average of 3.6 percent. It has engaged in a vigorous campaign to close schools — most serving low-income and minority students. Its pre-kindergarten program, while large, is of such poor quality that Steve Barnett of the National Institute for Early Education Research doubts it deserves the pre-k label.

Another factor distinguishing these states is the average income of their residents. In 2011, Connecticut ranked 4th of the 50 states, with a median household income of $65,753, and Massachusetts was next, at $62,859. Florida, in contrast, ranked 37th, at just $44,299. While high income itself does not drive scores, it reflects parents’ education and employment and a range of resulting public and private choices that interact with one another to strongly influence educational achievement. Moreover, these large resource disparities are compounded by states’ decisions regarding how they are used. Equitable school funding, which the PISA study finds to boost test scores, also varies greatly across states. The Education Law Center  puts Connecticut and Massachusetts slightly above average on equity, while Florida is one of only three states to receive poor grades on all four main indicators, making it among the very least equitable.

These findings shine a spotlight on the opportunity gaps driving achievement gaps across states (and within them). They thus make a strong case for a “Broader, Bolder Approach to Education” (a campaign of the Economic Policy Institute with which we are both affiliated); high-quality early childhood education, rigorous, enriching instruction for all students and comprehensive supports to help alleviate poverty-related impediments to learning. Equitable school funding can further narrow gaps. Education policies should not revolve around raising scores on PISA — or any test. But taking a page from the Massachusetts policy book — the broadest and boldest of any state — would improve the life prospects of disadvantaged students.

And if PISA scores go up too, that is certainly not a bad thing.