Should Scoring Rules be “Effective”?
Management Science · 1985 · Volume 31 · Issue 5
Abstract
A scoring rule is a reward function for eliciting or evaluating forecasts expressed as discrete or continuous probability distributions. A rule is strictly proper if it encourages the forecaster to state his true subjective probabilities, and effective if it is associated with a metric on the set of probability distributions. Recently, the property of effectiveness (which is stronger than strict properness) has been proposed as a desideratum for scoring rules for continuous forecasts, for reasons of “monotonicity” in keeping the forecaster close to his true probabilities, since in practice the forecast must be chosen from a low-dimensional set of “admissible” distributions. It is shown in this paper that what effectiveness implies, beyond strict properness, is not a monotonicity property but a transitivity property, which is difficult to justify behaviorally. The logarithmic scoring rule is shown to violate the transitivity property, and hence is not effective. The L 1 and L ∞ metrics are shown to allow no effective scoring rules. Some potential difficulties in interpreting admissible forecasts are also discussed.
Keywords
probability forecasting, evaluation of forecasts, proper scoring rules, effectiveness of scoring rules
Enable JavaScript for citation tools and additional paper interactions.