NAEP: Love It or Leave It!

As the national debate about what to do with education reform rages, I hear repeatedly the need to have national standards, common core standards, or at the very least, a more standardized system of national guidelines in order for us to measure and, presumably then, improve education in America. Often this discussion leads to debate about the merits of a national test—which is often assumed to be NAEP, sometimes called the "nation's report card.”

NAEP is well researched, well documented and seems to be well loved if not revered by most psychometricians—other than me and a few others who dare to challenge the status quo. I have questioned the usefulness of NAEP as a "check test" for NCLB at various times in my career, all based on the following premise:

• Students who take NAEP are essentially unmotivated; while most students are highly motivated to pass mandated state assessments.

• NAEP essentially measures a "consensus" national curriculum; while state assessments measure very specific content standards, which presumably align or mirror instruction.

• NAEP is individually administered via a specific student sample where students only take portions of the assessment; whereas all students take the complete statewide assessment.

• NAEP was targeted to measure at a higher level of proficiency (for example, 39% of all students were at or above proficient on NAEP Mathematics in 2007); whereas for most statewide assessments the percentages were much larger.

When I speak with my colleagues most of them say things like: Lighten up! NAEP is great! Math is math. Etc. Oddly enough, for a group of well-trained colleagues who claim to be scientists, such indefensible positions leave me wanting. Good news—there is some sensible research in the measurement literature. I am referring to the article by Dr. Andrew Ho, from the University of Iowa, published in Educational Measurement: Issues and Practices. In this article, entitled Discrepancies Between Score Trends from NAEP and State Tests: A Scale-Invariant Perspective, Dr. Ho provides a balanced and well-reasoned argument regarding the usefulness of NAEP for such comparisons. I provide quotes (albeit taken out of context) such that your curiosity will be peaked and you will seek out and read his entire article.

“Given a perspective that NAEP and State tests are designed to assess proficiency along different content dimensions, State-NAEP discrepancies are not cause for controversy but a baseline expectation.”

“Trends for a high-stakes, or ‘focal’ test, may differ from trends on a low-stakes test like NAEP (an ‘audit’ test) for a number of reasons, including different "elements of performance" sampled by both tests, different examinee sampling frames, or differing changes in student motivation.”

“As NAEP adjusts to its confirmatory role, there must be an additive effort to temper expectations that NAEP and State results should be identical.”

My colleagues, who have violated this last conclusion by Dr. Ho, are doing the policy makers and implementers of the NCLB "law of the land" a disservice by suggesting that statewide assessments are somehow inferior simply because their results are not replicated on NAEP. Let's get back to speaking about the science of assessment and experimental comparison and leave the passion and politics to someone else.