Paired comparison ratings: 2. The reliability of ratings based on partial pairings

Abstract

2 groups of 50 industrial employees were rated independently by their respective foremen using the method of paired comparison; all possible pairs of employees were rated. Various patterns of mutually-exculsive partial pairings were developed; in order to achieve mutually-exclusive pairings, 2 independent sets of pairs were developed for each pattern. The various patterns reduced from 24 to 8 the number of individuals with whom each employee in each independent set of pairs was paired; the total number of pairs for the various patterns ranged from 588 to 188 (a complete pairing resulting 1225 pairs). It was found that the reliability of ratings obtained with partial pairings tends to decrease rather systematically with reductions in the number of pairs per individual on which the ratings are based. For groups of 50 invividuals, ratings based on as few as 16 pairs per individual appear to be relatively reliable. (PsycInfo Database Record (c) 2025 APA, all rights reserved)