A rating-scoring method for free-response data

Abstract

A rating-scoring technique for evaluating free response answers is described and illustrated. An experimental (trainee) group of 18 supervisors and a matched control group of 18 each gave free response answers to four human relations questions. The E group was trained, and then both groups answered the questions again. Four social scientist raters sorted the 72 responses to each question into a 7-category forced-normal distribution. The pre and post-test score for each respondent was the sum of the category numbers assigned by the four raters over the four questions. Validity of the device as a measure of training effectiveness was previously reported (see 25: 7152). Interrater reliability is reported here: the summated pretest score reliability was .85; posttest score reliability was .88. (PsycInfo Database Record (c) 2025 APA, all rights reserved)