Read the latest published article from several of our research colleagues:
The Journal of Technology, Learning, and Assessment (JTLA)
Volume 10, Number 1, July 2010
The Effectiveness and Efficiency of Distributed Online, Regional Online, and Regional Face-to-Face Training for Writing Assessment Raters
Edward W. Wolfe, Staci Matthews, and Daisy Vickers
This study examined the influence of rater training and scoring context on training time, scoring time, qualifying rate, quality of ratings, and rater perceptions. 120 raters participated in the study and experienced one of three training contexts: (a) online training in a distributed scoring context, (b) online training in a regional scoring context, and (c) stand-up training in a regional context. After training, raters assigned scores to qualification sets, scored 400 student essays, and responded to a questionnaire that measured their perceptions of the effectiveness of, and satisfaction with, the training and scoring process, materials, and staff. The results suggest that the only clear difference on the outcomes for these three groups of raters concerned training time—online training was considerably faster. There were no clear differences between groups concerning qualification rate, rating quality, or rater perceptions.
Download the article (Acrobat PDF, 239 KB).
About The Journal of Technology, Learning and Assessment
The Journal of Technology, Learning and Assessment (JTLA) is a peer-reviewed, scholarly on-line journal addressing the intersection of computer-based technology, learning, and assessment. The JTLA promotes transparency in research and encourages authors to make research as open, understandable, and clearly replicable as possible while making the research process – including data collection, coding, and analysis – plainly visible to all readers.
