Cary Quattrocchi

The PISA methodology is complex and rather opaque, in spite of the substantial amount of material published in the technical reports. Briefly:

1. Individual students only answer a minority of questions.
2. Multiple ‘plausible values’ are then generated for all students assuming a particular statistical model, essentially estimating what might have happened if the student had answered all the questions.
3. These ‘plausible values’ are then treated as if they are the results of complete surveys, and form the basis of national scores (and their uncertainties) and hence rankings in league tables.
4. But the statistical model used to generate the ‘plausible scores’ is demonstrably inadequate – it does not fit the observed data.
5. This means the variability in the plausible scores is underestimated, which in turn means the uncertainty in the national scores is underestimated, and hence the rankings are even less reliable than claimed.

The comments on this page are not necessarily the views of Cary Quattrocchi

The average Node.js user is between 25-34 years old, according to the results.