Empirical Evaluation of Audio Encoders and Multimodal Fusion for Active Speaker Detection

Abstract

Active Speaker Detection (ASD) aims to identify the speaking individual within a visual scene across consecutive frames. This task has gained significant prominence due to its role in applications like Human-Robot Interaction (HRI). While existing multimodal approaches primarily focus on innovating video encoders, leveraging architectures such as Long Short-Term Memory (LSTM), Convolutional Neural Networks (CNN), and Graph Neural Networks (GNN), audio encoders have remained almost unaltered, often relying on Mel-frequency cepstral coefficients (MFCCs) processed through standard CNNs. In this paper, we explore the use of alternative audio encoding architectures and fusion mechanisms within Light-ASD framework. We conduct an extensive comparative analysis under in-database and cross-database settings using the AVA and WASD datasets. Furthermore, we investigate the impact of multi-dataset training by merging the training sets of both. Our results demonstrate that under in-database conditions, the MFCCs model achieves the best performance for AVA (93.54% mAP), while Wav2Vec yields the best results for WASD (94.28% mAP). In cross-database evaluations, X-Vectors exhibits the best generalization capabilities when trained on AVA (87.35% mAP, with a -2.76% relative performance drop relative to the in-database setting), whereas Wav2Vec generalizes more effectively when trained on WASD (83.62% mAP, with a -8.03% relative performance drop). Finally, we show that dataset fusion significantly enhances model robustness, delivering performance metrics that are comparable to their in-database setting, surpassing them in certain instances.