Quality Control for Crowd Workers and for Language Models: A Framework for Free-Text Response Evaluation with No Ground Truth
Inbal Yahav, Anat Goldstein, Tomer Geva, Shahar Meir, Onn Shehory
Information Systems Research · 2026 · Volume 37 · Issue 2
Abstract
In recent years, the field of natural language processing has made remarkable progress with the emergence of large language models (LLMs). In particular, the ability of LLMs to provide fact-based, free-text responses to user queries has the potential to revolutionize domains such as online search and the use of informative chatbots. However, extensive validation is required so that the response accuracy of question-answering LLMs can be confidently trusted. This paper introduces a framework to address this challenge: automated quality evaluation based on textual responses (AQER). The AQER framework focuses on two primary tasks: evaluating the quality of individual workers based on their free-text responses given that no ground-truth data are available and assessing the quality of LLM responses given a set of worker-generated responses. AQER is advantageously intuitive, easy to implement, and flexible to accommodate different components. To evaluate AQER’s effectiveness, we conducted empirical evaluations using semi-synthetic and real-world question-and-answer data sets as well as stress testing through numerical simulations. We also provide analytical motivation and show method convergence and boundary conditions using the probably approximately correct learning framework. The results demonstrate AQER’s robustness in evaluating LLMs and workers, and its superiority over baseline approaches. These findings establish AQER as a benchmark for future research in this field. History: Olivia Liu Sheng, Senior Editor; Dokyun (DK) Lee, Associate Editor. Funding: T. Geva and I. Yahav acknowledge research grants from the Jeremy Coller Foundation and the Henry Crown Institute for Business Research. Supplemental Material: The online appendix is available at https://doi.org/10.1287/isre.2023.0426 .
Keywords
worker evaluation, data science, machine learning, textual representation, large language models
Enable JavaScript for citation tools and additional paper interactions.