The Consequences of Consequential Validity

Recently I was in a meeting with some of our leaders in educational measurement when a discussion regarding how to present consequential validity evidence ensued. Having received most of my instruction, readings and otherwise general nurturing from several "disciples" of Bob Ebel, I was fascinated. I recalled the 1997 Educational Measurement: Issues and Practice volume (Vol. 16, No. 2, Summer) dedicated to this topic. An oversimplification of the debate, both then and now, seems to be around the definition of consequential validity itself. Bill Mehrens, in his 1997 article, articulates:

"I suggest that the psychometric community narrow the use of the term validity rather than expand it. Let us reserve the term for determining the accuracy of inferences about (and understanding of) the characteristic being assessed, not the efficacy of actions following assessment."

Professor Mehrens continues:

"The consequences of a particular use do not necessarily inform us regarding either the meaning of a construct or the adequacy of a particular assessment process in measuring that construct."

This group of measurement experts, to which I referred in the opening, debated such statements (and others) for quite some time. Having no resolution, but acknowledging that the consequences of the use of scores resulting from an assessment are important, they recommended that we follow the Standards regarding presentation of consequential validity evidence.

For the life of me, I can find no reference to general "consequential validity" in the Standards at all. I can find some references to "unintended consequences," but the notion of general consequences for test score use is not specifically addressed. Perhaps, even if it is only a memorialization of the debate, the next edition of the Standards will include it.