The Cost of Adding Parameters to a Model

Jeremy M. G. Taylor, Arminda L. Siqueira, Robert E. Weiss

Journal of the Royal Statistical Society: Series B (Statistical Methodology) · 1996 · Volume 58 · Issue 3

DOI: 10.1111/j.2517-6161.1996.tb02103.x

Abstract

For a general regression model withnindependent observations we consider the variance of the estimate of a quantity of interest under two scenarios. One scenario is where all the parameters are estimated from the data; the other scenario is where a subset of the parameters is assumed known at their true values and the remaining parameters are estimated. We focus on quantities of interest which are defined on the scale of the response variable. We show that, under certain conditions, the ratio of a weighted sum across the design points of the variance of the quantity of interest is given byq/p, whereqandpare the number of free parameters in the two scenarios. Thus, in this average sense, the inflation in variance associated with adding parameters, also interpreted as the cost of adding parameters to a model, is directly proportional to the number of parameters. We study models involving power transformations, non-linear models and exponential family models.

Keywords

box-cox transformations, conditional analysis, generalized linear model, location-scale model, non-linear regression, power transformations, prediction, unconditional analysis, variance inflation

Enable JavaScript for citation tools and additional paper interactions.