Nonparametric Analysis of Unbalanced Paired-Comparison or Ranked Data

Abstract

Suppose that we have t objects C 1 , …, C t and that C i and C j , are judged pairwise in n ij independent comparisons, for i, j = 1, …, t ; i ≠ j . In the simplest of such “paired-comparison” designs, all pairs of objects are compared an equal number of times (i.e., all n ij = n ). Yet it is often impractical to carry out such a “completely balanced” experiment: most observational data and even many planned experiments are highly unbalanced, and some are “incomplete” in that n ij = 0 for some pairs ( i, j ). Most of the available methods for analysis of unbalanced paired-comparison data are parametric, in the sense that a (paired-comparison) linear model generates, for each pair of objects, the “preference probability” π ij with which C i is preferred to C j . David (1987) proposed a simple nonparametric method of scoring objects from unbalanced paired-comparison data that takes into account differences in the strength of the opposition encountered by each object as well as possible differences in the { n ij }. In the present article, statistical properties of the proposed scores are developed for the general unstructured case and for special cases of partial balance, such as when objects are arranged in a group-divisible design. The asymptotic null distribution of these scores leads to a test for equality of the objects. Two small numerical examples illustrate the method. The approach is then extended to unbalanced ranked data. Existing nonparametric rank approaches fail to account adequately for differences in the strength of the opposition encountered. Numerical examples of unbalanced ranked data illustrate the comparison between the proposed method and a representative of the best of the available rank methods.