Spurious estimation of validity coefficients in composite samples: Some methodological considerations

Abstract

In much test validation research, composite validation samples are formed by aggregating subsamples containing predictor and criterion data from different geographic locations, different positions, or different jobs. Typically, the subsamples are treated as random partitions in spite of the fact that mean differences on the predictor, criterion, or both variables may exist. This article demonstrates that predictor or criterion mean differences may lead to over- or underestimation of the population validity coefficient, depending on the profile of mean differences on both variables. Referencing the aggregation bias literature, this article presents equations that explain the effect of mean differences across partitions of the validation sample in terms of a modification of the partial correlation. Solutions to the problem are discussed. (PsycInfo Database Record (c) 2025 APA, all rights reserved)