Interoperability: What a Concept

I am pleased to have the opportunity to contribute to the first True Scores blog of 2011. I thought a worthy topic to talk about is the concept of interoperability, which is quite a hot potato in assessment circles these days.

Many in the assessment world are waiting anxiously for the requests for proposals to develop the new Common Core assessments, and we know that both of the consortia funded by the federal government have expressed a commitment to interoperability in their submissions. Consider this excerpt from the proposal submitted by the Partnership for the Assessment of Readiness for College and Careers (PARCC):

“The Partnership’s proposed assessment system will take in and produce a wealth of data, and that information must seamlessly integrate with states’ data management systems. Currently, many state applications are closed off from each other and cannot easily integrate, extend or change as quickly as the requirements for assessment delivery, scoring and reporting change.”

A similar vision was espoused in the application submitted by the Smarter/Balanced Assessment Consortium (SBAC):

“The interoperable design will support a consistent system of assessment delivery across Consortium States, provide item portability as needed, ensure continuity of universal item design features across multiple platforms and uses, and provide competition among vendors for future work.”

These consortia implied that they and any vendors supporting the development of Common Core assessments could and would provide interoperable assessment items, tools, and systems that function at the cutting edge of technology and innovative assessment design. This is a compelling but somewhat unsettling vision, as I know that the new assessment tools my colleagues and I are developing extend well beyond the existing interoperability standards, such as the Question and Test Interoperability standard (QTI) from IMS Global Learning Consortium, and the Schools Interoperability Framework (SIF®) from the SIF Association.

To express our concerns, Pearson published a white paper on the topic of interoperability in late 2010. Our paper seems to have been timely, because a burst of activity regarding interoperability standards quickly followed. First, a meeting occurred in which the goals and progress of the Accessible Portable Item Profile (APIP) project were shared with the assessment community. This project, funded by a U.S. Department of Education Enhanced Assessment Grant, uses current QTI and Access for All specifications to develop an integrated set of tags, along with descriptions of expected behaviors, which can be applied to standardize the interoperability and accessibility of test items.

In late December, the U.S. Department of Education (USDE) issued a Request for Information (RFI) relating to assessment technology standards. They expect to use this information to help determine the appropriate interoperability standards for assessments and related work developed under the Race to the Top Assessment (RTTA) program.

Pearson was glad to see the USDE RFI, and our technology experts spent a lot of time thinking hard about the questions posed. We fully support the development of assessment technology standards and the goal of interoperable test content and assessment systems. But we also know that a lot of work will be needed to achieve the vision expressed by the Common Core assessment consortia.

In particular, an unprecedented need exists for collaboration between vendors and assessment specification governance boards to determine coherent and comprehensive assessment standards. Such standards must support the full life cycle of building, delivering, scoring, and reporting assessments and must address assessment item content, assessment assets, assessment item accommodations, assessment item metadata, test forms, scoring, reporting, and feedback. These standards must further verify that simultaneous goals of innovation and standardization are not at odds with one another, and that the process of extension does not constrict future growth.

Piece of cake, right? Well, we have to begin somewhere and the stakes are high enough that all of the stakeholders are incented to participate and cooperate. Let’s just hope we can get it right. Incidentally, Pearson has posted our response to the USDE RFI to our Next Generation Assessments website. Check it out if you want to read the details.

Walter (Denny) Way , Ph.D.
Senior Vice President, Psychometric & Research Services
Assessment & Information
Pearson