Trustworthy Multimodal Learning for Wearable Physiological Sensing

Venue: Knowledge-Based Systems (KBS) journal — under review. Role: Second author (Conceptualization, Investigation, Writing – review & editing). Seungpil Choi is first author; Eungjoo Lee is corresponding author.

Wearable physiological monitoring is increasingly used for stress and affect recognition — driver-state tracking, daily-life stress — but fusing heterogeneous sensor streams stays fragile: individual channels degrade or fail without warning, and most fusion models give no signal that anything went wrong.

This work pairs uncertainty-aware feature modulation with a post-hoc trustworthiness auditing mechanism. Modality-specific Transformer encoders are first aligned through auxiliary classification and mutual distillation, so the resulting representations support stable uncertainty estimation. Noise-perturbation sensitivity scores then down-weight unreliable modalities before a multimodal Transformer produces the fused prediction, with cyclic optimization and an EMA parameter update regularizing training dynamics.

At inference, the auditing scheme measures epistemic honesty under progressive sensor corruption and quantifies modality reliance under channel dropout — a sensor-level diagnostic of when and why the model’s outputs should be trusted. Across WESAD and AffectiveROAD the method reaches accuracy and F1 competitive with the compared approaches while additionally auditing its own uncertainty behavior; on WESAD, where sensor corruption is resolvable, it also spreads its reliance more evenly across sensors and responds more honestly to input degradation.

Uncertainty-aware feature modulation with post-hoc reliability auditing.