Abstract

Scene classification aims to identify scene categories by analyzing environmental information. In real-world scenes, scene features are primarily captured through acoustic and visual modalities. However, environmental complexity and information diversity pose significant challenges to classification performance. To address the issues of insufficient information interaction and inadequate feature fusion in traditional segmented fusion methods for audio–visual data, this paper proposes an audio–visual scene classification method based on a cross-modal attention mechanism. The fusion mechanism of multi-modal features is investigated, and scene classification performance is enhanced by optimizing the feature fusion strategy. The proposed method consists of three components: a cross-modal attention module, a gating unit, and a residual connection. The cross-modal attention module achieves adaptive feature alignment by establishing dynamic correlations between audio and visual features. The multi-modal gating unit employs an adaptive gating mechanism to dynamically adjust the contribution weight of each modality, thereby alleviating the information loss problem commonly encountered in traditional methods. The residual connection module preserves the original modality features to prevent information degradation. The model’s performance is evaluated through testing and validation on real-scene audio–visual datasets. Multiple sets of experimental results on these datasets demonstrate that the proposed cross-modal attention method achieves a significant improvement in classification accuracy.

IPC Classification

Keywords