Beware of snakes in the grass

Most people have a healthy fear of snakes. I don’t. I have an incredibly unhealthy fear of snakes. I see them where they are. I see them where they aren’t. I’ve been known to toss a bowl of popcorn into the air when they suddenly appear on the television screen. So, when I learned about the Cobra Effect, the circumstance in which a plan to reduce the venomous snake’s threat to humans actually increased the threat, I was horrified. But that’s nothing compared to the alarm I felt in reading The Testing Charade: Pretending to Make Schools Better in which Daniel Koretz uses the Cobra Effect as a mechanism to organize the junk drawer of problems with standardized testing in public education.

Specifically, Koretz outlines an example of the Cobra Effect known as Campbell’s Law in which social scientist Donald T. Campbell asserts “the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." Insufficient measures generate distortions, and Koretz argues that in education, standardized testing is the insufficient measure.

The distortions?

There are too many for a short review but Koretz does a good job organizing the basic misunderstandings, unintended consequences, and defeat devices associated with standardized testing. From the widening gaps among the intended, taught, and learned curricula… to the people-sorting nature of standardized tests… to the incentivizing of instructional behaviors that run counter to how people learn, Koretz reviews the specifics of how standardize testing is an incomplete instrument for evaluating teaching and learning.

More importantly, in using Campbell’s Law, Koretz includes education in a broader discussion about social change processes, cleverly applying in his thesis what he recommends for education evaluation; broaden the discussion, increase balance in the evaluation process, recognize that there are tradeoffs in managing complex systems, and stop trying to quantify everything.

As much as I would like to make the comparison, standardized tests are not the Cobras of this story. Unfortunately, the Cobras are the distortions and unintended consequences of their misuse; score inflation, test prep, the needlessly diminishing reputation of public education, the struggles of the poorest students in a system not designed to account for them, and the limitations of an evaluation instrument built around regression to the mediocre.

I won’t throw a bowl of popcorn into the air the next time I see a standardized test, but I do fear the real snakes in the grass, the unintended consequences of misusing standardized tests. The charade, as Koretz calls it, will eventually come back to bite us.