Speech signals contain rich information, such as textual content, emotion, and speaker identity. To extract these features more efficiently, researchers are investigating joint training across multiple tasks, like Speech Emotion Recognition (SER) and Speaker Verification (SV), aiming to improve performance by decoupling task-specific knowledge. Traditional multitask decoupling methods in SER typically use orthogonalization to increase the distance between parameter vectors in the feature space. In this paper, we introduce a novel Hybrid instance-level Contrastive Decoupling Loss. This method leverages supervised labels to effectively decouple SER and SV. Unlike previous approaches, it is not restricted to dual-stream models with identical architectures and can be easily integrated with leading models for each sub-task. Experimental results show that our proposed Hybrid Contrastive Learning Decoupling (HCLD) method significantly outperforms traditional orthogonal decoupling approaches.

Incorporating external knowledge has been shown to improve emotion understanding in dialogues by enriching contextual information, such as character motivations, psychological states, and causal relations between events. Filtering and categorizing this information can significantly enhance model performance. In this paper, we present an innovative Emotion Recognition in Conversation (ERC) framework, called the Scene-Speaker Emotion Awareness Network (SSEAN), which employs a dual-strategy modeling approach. SSEAN uniquely incorporates external commonsense knowledge describing speaker states into multimodal inputs. Using parallel recurrent networks to separately capture scene-level and speaker-level emotions, the model effectively reduces the accumulation of redundant information within the speaker’s emotional space. Additionally, we introduce an attention-based dynamic screening module to enhance the quality of integrated external commonsense knowledge through three levels: (1) speaker-listener-aware input structuring, (2) role-based segmentation, and (3) context-guided attention refinement. Experiments show that SSEAN outperforms existing state-of-the-art models on two well-adopted benchmark datasets in both single-text modality and multimodal settings.

In multimodal emotion recognition, disentangled representation learning method effectively address the inherent heterogeneity among modalities. To facilitate the flexible integration of enhanced disentangled features into multimodal emotional features, we propose a task-driven multimodal emotion recognition method TDMER. Its Cross-Modal Learning module promotes adaptive cross-modal learning of features disentangled into modality-invariant and modality-specific subspaces, based on their contributions to emotional classification probabilities. The Task-Contribution Fusion mechanism then assigns controllable weights to the enhanced features according to their task objectives, generating multimodal fusion features that improve the emotion classifier's discriminative ability. The proposed TDMER approach has been evaluated on two widely-used multimodal emotion recognition benchmarks and demonstrated significant performance improvements compared with other state-of the-art methods.

Multimodal emotion recognition (MER) is essential for understanding human emotions from diverse sources such as speech, text, and video. However, modality heterogeneity and inconsistent expression pose challenges for effective feature fusion. To address this, we propose a novel MER framework combining a Dynamic Weighted Graph Convolutional Network (DW-GCN) for feature disentanglement and a Cross-Attention Consistency-Gated Fusion (CACG-Fusion) module for robust integration. DW-GCN models complex inter-modal relationships, enabling the extraction of both common and private features. The CACG-Fusion module subsequently enhances classification performance through dynamic alignment of cross-modal cues, employing attention-based coordination and consistency-preserving gating mechanisms to optimize feature integration. Experiments on the CMU-MOSI and CMU-MOSEI datasets demonstrate that our method achieves state-of-the-art performance, significantly improving the 𝐴𝐶𝐶7 , 𝐴𝐶𝐶2, and 𝐹1 scores.

Speech emotion recognition (SER) poses one of the major challenges in human-machine interaction. We propose a new algorithm, the Voiced Segment Selection (VSS) algorithm, which can produce an accurate segmentation of speech signals. The VSS algorithm deals with the voiced signal segment as the texture image processing feature which is different from the traditional method. It uses the Log-Gabor filters to extract the voiced and unvoiced features from spectrogram to make the classification. The finding shows that the VSS method is a more accurate algorithm for voiced segment detection. Therefore, it has potential to improve performance of emotion recognition from speech.

Speech emotion recognition has been a prevalent research topic in recent years. Existing speech emotion recognition approaches mainly involve processing and analyzing speech signals, in order to discern the speaker’s emotions in speech. 2D Gabor filters have been used to extract the spectro-temporal features of the emotional information from spectrogram. We used Gabor filters to find and extract the major feature patterns for different emotions in spectrogram in our previous study which had concentrated on the parts of a sentence that demonstrated intensive expressions of emotions. In this paper, however, we further categorize emotional expressions in a sentence into primary and secondary ones according to its intensiveness and reveal the feature patterns of the secondary emotional expressions in spectrogram. We conducted feature extraction using Gabor filters on the feature patterns of both primary and secondary emotional expressions. Our experimental results outperformed those from the state-of-the-art and primary-patterns-focused algorithms. This demonstrates that secondary emotional feature patterns can be extracted and used to further improve the accuracy of speech emotion recognition.