When do those who know more also know more about how much they know? The development of confidence and performance in categorical decision tasks

Abstract

The study traces the development of confidence and its relation to performance as participants learn a new skill and develop from laymen to experts. We describe a new hierarchy of Categorical Decisions Structures that distinguishes between screening, discrimination, and classification modes. Each decision mode offers a different representation of information and provides a different type of feedback. We propose simple models of performance and confidence, and derive their predictions under all three modes. We report results of an empirical study designed to evaluate the models’ predictions. Ninety participants performed 1200 categorical decisions under the three modes and three distinct base rates. Results show that (a) decisions made in the screening mode tend to induce over-confidence; (b) the discrimination mode leads to fast learning and high correspondence between performance and confidence; and (c) the detailed feedback provided in the classification mode results in slow and steady improvement of the correspondence between confidence and performance.