Variations on a seminal demonstration of people's insensitivity to sample size

Abstract

Most people do not seem to realize that an extreme value for a sample statistic is more likely the smaller the sample. However, Harvard undergraduates, who have satisfied a Quantitative Reasoning Requirement, do much better on some versions of sample size questions than previous populations have. Nevertheless, these results show that subjects' difficulty in selecting the correct answer to questions about the relationship between sample statistic and sample size is not related to the presence or wording of an answer choice that states there is no relationship. Further, formally identical versions of problems yield quite different rates of correct answers.