Information gain and a general measure of correlation

Abstract

Given a parametric model of dependence between two random quantities, X and Y , the notion of information gain can be used to define a measure of correlation. This definition of correlation generalizes both the usual product-moment correlation coeffi cient for the bivariate normal model and the multiple correlation coefficient in the standard linear regression model. The use of this information-based correlation in a descriptive statistical analysis is examined and several examples are given.