Higher criticism for discriminating word-frequency tables and authorship attribution
The Annals of Applied Statistics · 2022 · Volume 16 · Issue 2
Abstract
We adapt the higher criticism (HC) goodness-of-fit test to measure the closeness between word-frequency tables. We apply this measure to authorship attribution challenges, where the goal is to identify the author of a document using other documents whose authorship is known. The method is simple yet performs well without handcrafting and tuning, reporting accuracy at the state-of-the-art level in various current challenges. As an inherent side effect, the HC calculation identifies a subset of discriminating words. In practice, the identified words have low variance across documents belonging to a corpus of homogeneous authorship. We conclude that in comparing the similarity of a new document and a corpus of a single author, HC is mostly affected by words characteristic of the author and is relatively unaffected by topic structure.
Keywords
authorship attribution, Feature selection, higher criticism, nonparametric methods, two-sample testing, unsupervised learning
Enable JavaScript for citation tools and additional paper interactions.