Improving the measurement of group-level constructs by optimizing between-group differentiation

Abstract

The ability to detect differences between groups partially impacts how useful a group-level variable will be for subsequent analyses. Direct consensus and referent-shift consensus group-level constructs are often measured by aggregating group member responses to multi-item scales. We show that current measurement validation practice for these group-level constructs may not be optimized with respect to differentiating groups. More specifically, a 10-year review of multilevel articles in top journals reveals that multilevel measurement validation primarily relies on procedures designed for individual-level constructs. These procedures likely miss important information about how well each specific scale item differentiates between groups. We propose that group-level measurement validation be augmented with information about each scale item’s ability to differentiate groups. Using previously published datasets, we demonstrate how ICC(1) estimates for each item of a scale provide unique information and can produce group-level scales with higher ICC(1) values that enhance predictive validity. We recommend that researchers supplement conventional measurement validation information with information about item-level ICC(1) values when developing or modifying scales to assess group-level constructs. (PsycInfo Database Record (c) 2025 APA, all rights reserved)