Abstract

This study compares the effectiveness of two approaches to speech emotion recognition for three affective states in Polish: sad, neutral, and happy. Both a set of acoustic features—capturing prosodic, phonatory, temporal, spectral, and cepstral properties—and representations learned by self-supervised models (wav2vec 2.0 and WavLM) were analyzed. Experiments were conducted on the nEMO corpus, comprising 2327 recordings from nine speakers, using a rigorous leave-one-subject-out protocol to evaluate cross-speaker generalization. In the feature-based approach, 107 acoustic features were used, and classification was performed with logistic regression and, additionally, SVM variants. In the deep learning approach, the wav2vec2-base and WavLM-base models were fine-tuned for the three-class task. The best results were achieved by the self-supervised models: WavLM reached a global balanced accuracy of 0.727 and a macro-F1 score of 0.710, while wav2vec 2.0 achieved 0.722 and 0.695, respectively. Both outperformed the feature-based approach (BAcc = 0.627, macro-F1 = 0.584). Confusion matrix analysis showed that the greatest difficulty lies in distinguishing the neutral class from the sad and happy classes, whereas sad and happy classes are more clearly separable. Feature utility analysis (SFS under the LOSO protocol) indicated the significant role of cepstral features (MFCCs and their derivatives), complemented by selected prosodic and temporal features. An additional comparison of SVM classifiers suggested that the main limitation of this approach lies in the signal representation itself rather than solely in the choice of classifier. Explainability analyses of the deep models, using layer-wise probing and integrated gradients, showed that affective information is best represented in intermediate layers, and that model decisions rely on locally salient segments of the signal. Furthermore, a speaker adaptation experiment demonstrated that personalization significantly improves classification performance, highlighting the potential of such methods for long-term monitoring of affective expression changes in the same individual.

IPC Classification

Keywords