Word-Level Maximum Mean Discrepancy Regularization for Word Embedding

Journal of the American Statistical Association · 2026 · Volume 121 · Issue 553

DOI: 10.1080/01621459.2025.2547978

Abstract

The technique of word embedding is widely used in natural language processing (NLP) to represent words as numerical vectors in textual datasets. However, the estimation of word embedding may suffer from severe overfitting due to the huge variety of words. To address the issue, this article proposes a novel regularization framework that recognizes and accounts for the “word-level distribution discrepancy”—a common phenomenon in a range of NLP tasks where word distributions are noticeably disparate under different labels. The proposed regularization, referred to as word-level MMD ( wMMD ), is a variant of maximum mean discrepancy (MMD) that serves a specific purpose: to enhance/preserve the distribution discrepancies within word embedding numerical vectors and thus prevent overfitting. Our theoretical analysis illustrates that wMMD can effectively operate as a dimension reduction technique of word embedding, thereby significantly improving the robustness and generalization of NLP models. The numerical effectiveness of wMMD and its variants is demonstrated in various simulated examples, CE-T1 and BBC News datasets with state-of-the-art NLP deep learning architectures. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Keywords

Covering number, Generalization bounds, Maximum mean discrepancy (MMD), Statistical regularization, Word embedding

Enable JavaScript for citation tools and additional paper interactions.