.
How can you be confident that local officials didn't pull any fast ones with their PISA results? Well, you can't, but you can get some sense of how much room there is to pull the wool over your eyes by looking at the response rate.
Large countries have to test at least 4,500 students, and the sample is supposed to be carefully designed to represent the entire country's 15-year-olds. But projected coverage usually turns out less than perfect. For example, countries can exclude students with disabilities. This sounds reasonable -- it's hard for a blind person to take a pencil and paper test. But, what about cognitive disabilities, such as not being very bright? From the federal government's website on PISA:
PISA 2012 is designed to be as inclusive as possible. The guidelines allowed schools to be excluded for approved reasons (for example, schools in remote regions, very small schools, or special education schools). Schools used the following international guidelines on student exclusions:
Students with functional disabilities. These were students with a moderate to severe permanent physical disability such that they cannot perform in the PISA testing environment.
Students with intellectual disabilities. These were students with a mental or emotional disability and who have been tested as cognitively delayed or who are considered in the professional opinion of qualified staff to be cognitively delayed such that they cannot perform in the PISA testing environment.
Students with insufficient language experience. These were students who meet the three criteria of not being native speakers in the assessment language, having limited proficiency in the assessment language, and having less than 1 year of instruction in the assessment language.
Overall estimated exclusions (including both school and student exclusions) were to be under 5 percent of the PISA target population.
Buried in a PISA appendix entitled Annex 2A are PISA figures for what percentage of the target populations of 15-year-olds didn't get tested. America didn't come close to getting 95% representation, and many Third World countries were far worse.
"Coverage Index 3: Coverage of 15-year-old population" shows what percentage of the cohort are represented if the test taking sample was projected to the whole country. I subtracted this percentage from 100% to come up with the % Missing index. For example, Costa Rica only managed to test half the people they were supposed to, and Albania only tested 55%. Vietnam, which made a splashy PISA debut with high scores, somehow couldn't find 44% of their 15-year-olds. At the other end, the fastidious Dutch managed to test slightly more students than were thought to be around.
| % Missing | |
| Costa Rica | 50% |
| Albania | 45% |
| Vietnam | 44% |
| Mexico | 37% |
| Colombia | 37% |
| Indonesia | 37% |
| Turkey | 32% |
| Brazil | 31% |
| Thailand | 28% |
| Peru | 28% |
| Uruguay | 27% |
| Liechtenstein | 25% |
| Bulgaria | 23% |
| Shanghai-China | 21% |
| Malaysia | 21% |
| Argentina | 20% |
| Kazakhstan | 19% |
| Macao-China | 19% |
| Hungary | 18% |
| United Arab Emirates | 17% |
| Canada | 17% |
| Chile | 17% |
| Hong Kong-China | 16% |
| Czech Republic | 15% |
| Serbia | 15% |
| Latvia | 15% |
| Lithuania | 14% |
| Jordan | 14% |
| Australia | 14% |
| Italy | 14% |
| Greece | 13% |
| New Zealand | 12% |
| Korea | 12% |
| Austria | 12% |
| Portugal | 12% |
| Spain | 12% |
| France | 12% |
| United States | 11% |
| Chinese Taipei | 11% |
| Poland | 11% |
| Luxembourg | 11% |
| Montenegro | 10% |
| Israel | 9% |
| Denmark | 9% |
| Japan | 9% |
| Ireland | 9% |
| Slovak Republic | 9% |
| Tunisia | 9% |
| Switzerland | 9% |
| Norway | 8% |
| Estonia | 8% |
| Russian Federation | 8% |
| Iceland | 7% |
| Sweden | 7% |
| United Kingdom | 7% |
| Slovenia | 6% |
| Qatar | 6% |
| Croatia | 6% |
| Germany | 5% |
| Singapore | 5% |
| Belgium | 5% |
| Finland | 4% |
| Romania | 4% |
| Cyprus | 3% |
| Netherlands | -1% |
In general, Third World countries were bad at getting good coverage, suggesting that the First World v. Third World gap is even larger than the test scores imply.
Top scorer Shanghai missed 21%, so we should take its flashy scores with a few grains of salt.
America was at 11% missing, down from 18% missing in 2009, which may account for the slight decline in U.S. scores?
Consistent high-flier Finland had only 4% missing, so they aren't cheating on this measure more than the competition is.
Top scorer Shanghai missed 21%, so we should take its flashy scores with a few grains of salt.
America was at 11% missing, down from 18% missing in 2009, which may account for the slight decline in U.S. scores?
Consistent high-flier Finland had only 4% missing, so they aren't cheating on this measure more than the competition is.
A major question is how random were the missing test-takers. If the missing were purely random, then no harm no foul. But of course, many of the missing are dropouts, or in special day classes, or in juvy hall, or whatever.
This may help excuse slightly Argentina's horrible scores. The Argentineans misplaced only 20% of their 15-year-olds compared to the 37% of Mexicans who went missing.
Printer-friendly version