Meta Fusion: a unified framework for multimodal fusion with adaptive mutual learning

Journal of the Royal Statistical Society: Series B (Statistical Methodology) · Thu

Abstract

Developing effective multimodal data fusion strategies has become increasingly essential across a wide range of applications, from autonomous driving to medical diagnosis. Traditional fusion methods, including early, intermediate, and late fusion, integrate data at different stages, each offering distinct advantages and limitations. In this article, we introduceMeta Fusion, a flexible and principled framework that unifies these strategies as special cases. Motivated by deep mutual learning and ensemble learning, Meta Fusion constructs a cohort of models based on various latent representation combinations, and further boosts predictive performance throughsoft information sharingwithin the cohort. Our approach is model-agnostic in learning the latent representations, allowing it to flexibly adapt to the unique characteristics of each modality. Theoretically, our soft information sharing mechanism reduces the generalization error. Furthermore, we develop an uncertainty quantification framework that provides rigorous coverage guarantees with minimal assumptions. Empirically, Meta Fusion consistently outperforms conventional strategies across extensive numerical studies, including Alzheimer’s disease detection and neural decoding.

Keywords

deep mutual learning, ensemble selection, multimodality fusion, representation learning, soft information sharing, uncertainty quantification

Enable JavaScript for citation tools and additional paper interactions.