BCN Perceptual Computing Lab Repository

Modelling and Analyzing Multimodal Dyadic Interactions Using Social Networks

Escalera, Sergio and Radeva, Petia and Vitrià, Jordi and Baró, Xavier and Raducanu, Bogdan (2010) Modelling and Analyzing Multimodal Dyadic Interactions Using Social Networks. Proceedings of the 12th International Conference on Multimodal Interfaces and 7th Workshop on Machine Learning for Multimodal Interaction (ICMI-MLMI 2010) .

[img]PDF (Paper) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
[img]Microsoft PowerPoint (Slides)


Social network analysis became a common technique used to model and quantify the properties of social interactions. In this paper, we propose an integrated framework to explore the characteristics of a social network extracted from multimodal dyadic interactions. First, speech detection is performed through an audio/visual fusion scheme based on stacked sequential learning. In the audio domain, speech is detected through clusterization of audio features. Clusters are modelled by means of an One-state Hidden Markov Model containing a diagonal covariance Gaussian Mixture Model. In the visual domain, speech detection is performed through differential-based feature extraction from the segmented mouth region, and a dynamic programming matching procedure. Second, in order to model the dyadic interactions, we employed the Influence Model whose states encode the previous integrated audio/visual data. Third, the social network is extracted based on the estimated influences. For our study, we used a set of videos belonging to New York Times’ Blogging Heads opinion blog. The results are reported both in terms of accuracy of the audio/visual data fusion and centrality measures used to characterize the social network.

Item Type:Article
Subjects:Q Science > QA Mathematics > QA75 Electronic computers. Computer science
ID Code:50
Deposited By:Xavier Baró
Deposited On:02 Dec 2010 13:07
Last Modified:02 Dec 2010 13:07

Repository Staff Only: item control page