The impact of ratee performance characteristics on rater cognitive processes and alternative measures of rater accuracy

Abstract

In a repeated measures design, the effects of two ratee performance characteristics (performance level and performance consistency) on rater cognitive processes and rating accuracy were examined. Thirty-seven raters evaluated the performance of four subordinates who differed from each other on their performance level and consistency. It was hypothesized that these performance characteristics would influence the amount of performance information collected by raters as well as the way in which they processed this information. In addition, it was suggested that performance characteristics would differentially affect several types of rating accuracy measures. Cronbach's elevation accuracy and two indices of accuracy based on signal detection theory, labeled behavioral and classification accuracy by R. G. Lord (1985, Journal of Applied Psychology, 70, 66–71), were compared. Results showed that characteristics of ratee performance influenced both rater cognitive processes and rating accuracy.