The use of levels of confidence in item analysis

Abstract

Using the RBH Supervisory Judgment Test to predict an intelligence test criterion (AFQT), an experimental population was divided into 3 samples (N of 80, 150, 300), and 4 keys were constructed for the SJT for each sample. The keys included those items that differentiated between the upper and lower 27% of the AFQT distributions at the 1%, 5%, 20%, and 50% levels of confidence respectively. The number of items included increased with both increasing size of sample and level of confidence. Cross-validation of the 12 keys on the original Ss (redistributed) showed that there were no significant differences in total score validity attributable to sample size or level of confidence. "The results… indicate that there is no one optimal level of confidence which should be employed when analyzing test data." (PsycInfo Database Record (c) 2025 APA, all rights reserved)