Empirical Bayes and Item-Clustering Effects in a Latent Variable Hierarchical Model

Abstract

Empirical Bayes regression procedures are often used in educational and psychological testing as extensions to latent variables models. The National Assessment of Educational Progress (NAEP) is an important national survey using such procedures. The NAEP applies empirical Bayes methods to models from item response theory to calibrate student responses to questions of varying difficulty. Due partially to the limited computing technology that existed when NAEP was first conceived, NAEP analyses are carried out using a two-stage estimation procedure that ignores uncertainty about some model parameters. Furthermore, the item response theory model that NAEP uses ignores the effect of item clustering created by the design of a test form. Using Markov chain Monte Carlo, we simultaneously estimate all parameters of an expanded model that considers item clustering to investigate the impact of item clustering and ignoring uncertainty about model parameters on an important outcome measure that NAEP report. Ignoring these two effects causes substantial underestimation of standard errors and induces a modest bias in location estimates.