Policy Wonks We Are—Implications for NCME Members

This is the "full text" of a contribution I made to the NCME newsletter. I thought you might like to get the full inside story!

Mark Reckase’s call for NCME members to become more involved in educational policy is timely and relevant, while perhaps also a little misleading. For example, some of my colleagues and I have been working with states, local schools, and the USDOE regarding implementing policy decisions for many years. Testifying at legislative hearings, making presentations to Boards of Education, reviewing documents like the Technical Standards, and advising policy makers are all examples of how psychometricians and measurement experts already help formulate and guide policy. Nonetheless, I still hear many members of technical advisory committees (experts in psychometrics and applied measurement) “cop out” when asked to apply their experience, wisdom, and expertise to issues related to education policy, often citing that they are technical experts and the question at hand is “a matter of policy.”

I have commented and I believe that we no longer live in a world where the policy and technical aspects of measurement can remain independent. In fact, some good arguments can be made that when such independence (perhaps bordering on isolation) between policy and good measurement practice exists, poor decisions can result. When researchers generate policy governing the implementation of ideas, they must carefully consider a variety of measurement issues (e.g. validity, student motivation, remediation, retesting, and standard setting) to avoid disconnects between what is arguably good purpose (e.g. the rigorous standards of NCLB) and desired outcomes (e.g. all students meeting standards).

In this brief text, I will entertain the three primary questions asked by Dr. Reckase: (1) Should NCME become more involved in education policy? Why or why not? (2) How should other groups and individuals in the measurement community be involved in education policy? (3) What resources and supports are necessary to engage measurement professionals in education policy conversations? In what ways should NCME be involved in providing these?

I think I have already answered the first question, but let me elaborate. I maintain that we measurement professionals are already involved in policy making. Some of us influence policy directly (as in testifying before legislatures developing new laws governing education). Some of us influence policy in more subtle ways, by researching aspects of current or planned policy we do not like or endorse. We often seek out the venue of conference presentations to voice our opinions regarding what we think is wrong with education and how to fix it, which inevitably means we make a policy recommendation.

Not only do I believe that NCME and its members are involved in policy making, but I also believe it is critically important for all researchers and practitioners in the measurement community to seek out opportunities to influence relevant policy. I recall recently being involved in some litigation regarding the fulfillment of education policy and the defensibility of the service provider’s methods. After countless hours of preparation, debate, deposition, and essentially legal confrontation, I asked my colleague (also a measurement practitioner) why we bother defending best practice when there are so many agendas, so many different ways to interpret policy, so many points of view regarding the “correct way” to implement a measure. Her response was surprising—she said we do it because it is the “right thing to do” and that if we stop defending the right way to do things, policy makers will make policy that is convenient but not necessarily correct. Her argument was not about defining right from wrong; her argument was that if we were not there instigating debate there would be none, and resulting decisions would most likely be poorly informed.

So, my simple answer to the second question is to get involved. If you don’t like NCLB, what did you do to inform the policy debate before it became the law of the land? If you think current ESL, ELL, or bilingual education is insufficient to meet the demands of our ever-increasing population in these areas, what are you doing to help shape policy affecting them? Across the country debate rages regarding the need for “national standards” or state-by-state comparability. Why aren’t NCME, AERA, and all other organizations seemingly affected by such issues banding together to drive the national debate? Do we not all claim to be researchers? If so, is not an open debate what we want and need? When was the debate where it was decided that the purpose of a high school diploma was college readiness? When did we agree to switch the rhetoric from getting everyone "proficient" by 2014 to getting everyone “on grade level” by 2014? The input of measurement experts was sorely missing in state legislation regarding these issues. It is still desperately needed.

For the purpose of this presentation, let’s assume that all measurement and research practitioners are in agreement that we need to take part in policy discussions directly. What resources, tools, and/or procedures can we use to implement these discussions and how can NCME help? I stipulate that there is a feeling of uneasiness surrounding the engagement of researchers and measurement practitioners in policy debates or decisions.

Perhaps this is an unfounded concern, but there seems to be an air, forgive me, of such debates being below our standards of scientific research. Policy research is very difficult (to generate and to read), so why leave the comforts of a safe “counter-balanced academic research design” to mingle with such “squishy” issues as the efficacy of policy implementation? Perhaps NCME could strive for a division or subgroup on Federal and State Policy that would focus on measurement research as it applies to education policy (policy, law making, and rule implementation) to lend more credibility to such a scientific endeavor. Maybe NCME could work with other groups with similar interests (like AERA, ATP, CCSSO) and maybe even get a spot in the cabinet of the next Secretary of Education for the purpose of promoting the credibility of measurement research and application for informing policy. Perhaps less ambitious things like including more policy research in measurement publications, sponsoring more policy discussions and national conventions, and encouraging more policy-related coursework in measurement-related Ph.D. programs would be a good place for NCME (and other organizations) to start.

Let me close with a simple example of why this interaction between applied measurement and education policy is so important. Many of you are firm believers in the quality of the NAEP assessments. Some of you have even referred to NAEP as the “gold standard” for assessment. NAEP is arguably the most researched and highest quality assessment system around. Yet, to this day many of my customers (typically the educational policy makers and policy implementers in their states) ask me simple questions: Why is NAEP the standard of comparison for our NCLB assessments? NAEP does not measure our content standards very well; why are our NAEP scores being scrutinized? What research exists demonstrating that NAEP is a good vehicle to judge education policy—both statewide and for NCLB?

Don’t get me wrong. My argument here is not against NAEP or the concept of using NAEP as a standard for statewide comparability. My question is why my customers—the very people making educational policy at the state level—were not at the table when such issues were being debated and adopted? Did such a debate even take place? As measurement experts, when our customers come to us for advice or guidance, or with a request for research regarding the implementation of some new policy, I believe it is our obligation to know and understand the implications of such a request from a policy point of view, not just a measurement point of view. Otherwise, we will be acting in isolation and increasing the divide between sound measurement practice and viable educational policy.