Agreement between expert-system and human raters' scores on complex constructed-response quantitative items

Abstract

Evaluated agreement between expert-system and human scores on 12 constructed-response algebra word problems taken by Graduate Record Examination General Test examinees. Problems were drawn from 3 content classes (rate by time, work, and interest) and presented in 4 constructed-response formats (open ended, goal specification, equation setup, and faulty solution). Agreement was evaluated for each item separately by comparing the system's scores to the mean scores taken across 5 content experts. The expert system produced scores for all responses and duplicated the judgments of raters with reasonable accuracy; the median of 12 correlations between the system and human scores was .88, and the largest average discrepancy was 1.2 on a 16-point scale. No obvious differences in scoring agreement between constructed-response formats or content classes emerged. Ideas are discussed for using expert scoring systems in large-scale assessment programs and in interactive diagnostic assessment. (PsycInfo Database Record (c) 2025 APA, all rights reserved)