Commonality in Assessment Solutions

Why isn’t it simple to develop common (off-the-shelf) solutions for processing assessments? I have come to realize that this industry does have some very subtle (and some not so subtle) complexities. Is all of this complexity really necessary?

I see two levels of complexity in assessment processing:
1. Developing, processing, scoring, and reporting assessments themselves, and
2. Handling the variability of data and business rules to process the data across programs.

The first level of complexity is mostly unavoidable. Variability at this level should be entirely driven by the assessment design, purpose, and types of items being used. However, the second level of complexity is mostly avoidable if we could agree on common structures and rules for defining and processing data.

Recent activities in Race to the Top and Common Core State Standards are exploring the use of “open industry standards” and “open source” to drive assessment solutions design and development. To that I say, “Outstanding!” However, are we thinking too narrowly? The standardization efforts have initially focused on learning standards, assessment content, and assessment results standards – again, all good. But what about all the data and processes surrounding the assessments? We will continue to develop custom data, business rules, and processes for each implementation of the “common” assessments to deal with the second level of variability.

Much of this variability is not a result of “adding value” to the assessment design or the outcomes of the assessment but has more to do with accommodating existing data management systems and processes, or in some cases simple preferences. While most of this may fall into the category of “just being flexible,” there are thousands of examples of variability in assessment solutions. These extreme levels of variability (or flexibility) have directly contributed to a very high level of custom software development in our industry. For an industry that is very schedule-, cost-, and quality-sensitive, this reality seems counterintuitive.

Here is a (non-comprehensive) list of potential opportunities for commonality:

• Matching rules and assignment of state student identifiers


• District and school identifiers, structures, and data


• Scoring rules such as best form, attemptedness, and rules for scoring open-ended items


• Multiple test attempt processing and status codes


• Scanable form design


• Rules and processes for packaging and distributing testing materials


• Report designs

If the efforts associated with Common Core could consider much greater commonality across implementations, vendors could provide more consistent “off-the-shelf” solutions for all states. The solutions also become highly reliable with less customized code, data, and processing rules being injected into every customer’s deployment. Unique custom solutions – as experience tells us – can be more costly and more prone to schedule and quality issues.

Assessment industry examples that illustrate the potential of common capabilities include the American Diploma Project (ADP), ACT and SAT, Iowa Test of Basic Skills (ITBS), and Stanford Achievement Tests (SAT10), among many other catalogue programs. These programs provide highly common data, processes, and reporting engines for all states or districts using them. Some variability is possible but within well defined constraints. While these programs have felt the pressure to customize, they have been able to retain many of their core data and capabilities. For example, all members of the ADP consortium agreed on demographic reporting categories for the Algebra assessment; therefore, each state collects and reports demographic data in the same way for the same assessment.

So now you might be thinking “talk about stifling innovation!” Actually, I am suggesting that we stifle variability where it does not add value to the assessment design or outcome. Instead, invest all of that energy in innovation for assessment instruments and interpreting results to improve teaching and learning.

As a final note, I have been careful to use the term “common” versus “standard.” It is possible to obtain commonality without formal standards – fully recognizing that commonality can also be achieved through the implementation of formal standards. All the standards development efforts in this industry are moving us closer to commonality, but are they timely enough and can the multiple standards converge?

Work will begin very soon on the Common Core assessment content and supporting platform designs, most likely in advance of the formal industry standards being available to support all activities. By working with the vendors who provide assessment services, it may be possible to make common all of those processes which are not directly contributing to improving teaching and learning - thus simplifying solutions, reducing costs, and allowing that money to be saved or diverted to high value innovations.

Complementary or contradictory points of view are welcomed.

Wayne Ostler

Vice President Technology Strategy
Data Solutions
Assessment & Information
Pearson