Item Difficulty, Discrimination, and the Confidence-Frequency Effect in a Categorical Judgment Task

Abstract

Three experiments are presented to examine the role of discrimination abilities in the relationship between confidence and performance across items that vary in difficulty. The studies also test the confidence-frequency effect, predicted in the theory of probabilistic mental models (Gigerenzer, Hoffrage, & Kleinbolting, 1991), by investigating the relationship between performance estimates provided as confidence judgments and as estimates of the frequency of correct responding. The categorical judgment task involved predicting whether a handwriting sample, generated using naturalistic sampling, had been written by a female or male. The results suggest that, when outcome variability is taken into account, discrimination ability differs drastically between easy, medium, and hard items. Discriminability is greatest for easy items and decreases until it disappears for hard items. There was also evidence of a confidence-frequency effect, with mean confidence judgments showing a slight tendency toward overconfidence and frequency estimates showing a slight tendency to underestimate performance.