Testing for agreement between two groups of judges

Abstract

The ‘problem of m rankings’, so named by M. G. Kendall and studied extensively by Kendall & Babington Smith (1939), Kendall (1970) and others, considers the relationship between the rankings that a group of m judges assigns to a set of k objects. Suppose that there are two groups of judges ranking the objects. Given that there is agreement within each group of judges, how can one test for evidence of agreement between the two groups ? Schucany & Frawley (1973) proposed a test of agreement between the groups, and Li & Sohucany (1975) advanced this further. We show that this test must be used with care if the relevant hypothesis is to be taken as the null hypothesis. We then adapt a procedure, proposed by Wald & Wolfowitz (1944) in a slightly different context, to furnish a new test for agreement between two groups of judges