Review of Difficulty prediction of test items

Abstract

Reviews the book "Difficulty prediction of test items" by S. Tinkleman (see record 1948-04723-001 ). Using a civil service examination for patrolman consisting of 100 5-option multiple choice items covering arithmetic reasoning, civic information, logical inference, reading comprehension, judgment, vocabulary and other topics, the author had thirty judges rate the difficulty of each item. This rated difficulty (per cent of patrolman candidates expected to answer correctly) was compared with the actual percent of patrolman candidates passing. The validity of the judges' estimates of difficulty ranged from .23-.77 with a median of .53. The validity of the pooled judgment of the thirty judges was .76. Even within the limits of "validity" represented by a median coefficient of .53, there were large constant errors of over- or underestimation. The author notes that the judges tended to overestimate the percentage of passes on difficult items and to underestimate it for easy items. The judgments thus exhibited the usual regression toward the mean. The tone of the conclusions and implications seems unwarrantedly optimistic. "The results of this study indicate that judges are probably able to estimate relative item difficulty accurately." This conclusion does not seem warranted for a median validity correlation of .53. Although Tinkelman's book does shed light on the process of judging difficulty, it will not convince the careful examiner that estimated difficulty is an adequate substitute for empirical data. (PsycInfo Database Record (c) 2025 APA, all rights reserved)