Rater reliability and the heterogeneity of scale anchors

Abstract

"Raters (N=200) used 5-category rating scales, which varied in the heterogeneity of the verbal anchors defining the end categories, to rate for preference value two lists of ten foods. The food lists differed in the homogeneity of food stimuli on each list. Measures of individual rater reliability and rater bias were computed and analyzed as to the effect of scale and list differences. Reliability was significantly smaller for the more homogeneous list and increased linearly as a function of the heterogeneity of the end anchors. Rater bias was unaffected by either scale or list differences." (PsycInfo Database Record (c) 2025 APA, all rights reserved)