Cross-Modal Coordination in Behavioral and Physiological Synchrony

Recent research in Behavioral and Physiological Synchrony underlie several prosocial behaviors, including co-sleeping, narrative processing in attentive listening, and shared attention in conversations. With constantly improving data collection tools offered by modern technology, we are primed to better understand the relationships between different behavioral (e.g. acoustic prosody in voice, head and body pose) and physiological signals (heart rate variability, eye-tracking) to describe biobehavioral states. Developments on this end will support both better understanding of within person dynamics of multimodal behavior, and coordinating dynamics of multimodal behavior between individuals. We are beginning to see the development of multimodal behavioral and physiological interfaces that permeate our daily lives, from driver behavior tracking for in-vehicle safety systems, to video conferencing technologies designed to mimic in-person interactions, to collaborative stress detection in teams to promote wellbeing in the workplace.

Such innovations are still dependent on analytical improvements in methodology for characterizing multimodal/cross-modal coordination in behavioral and physiological synchrony. Human behavior is implicitly multimodal by nature. It thus precipitates that the most reliable indicators of biobehavioral states and changes are also multimodal by nature. Though these states can be measured by individual modalities, robust indicators are most likely to come from cross-modality fusion of signals.

My research focuses on developing analytical methods for measuring cross-modal signals. This begins with “classical” methods such as cross-sample entropy, to characterize a quasi-ground truth relationship between cross-modal signals, and to investigate the mechanistic effects governing their interactions. For example, we may determine the directionality of information transfer from a physiological signal to the behavioral signal via granger causality analysis. These methods, when used properly, are shown to provide excellent insight into the underlying processes which govern biobehavioral states.

Despite the success of classical methods, there are still limitations in measurement that need to be addressed. Latent variable modeling can be utilized to overcome such limitations, especially in data where there are known confounding effects on signals. For example, heart rate variability signals are highly susceptible to confounders such as age. Latent variable modeling via methods including higher order tensor factorization and variational inference, may enable better measurement of cross-modal signal coordination.

Currently, I am studying cross-modal coordination of head pose behaviors and heart rate variability in newly formed couple dyads. In our study, dubbed the “Aron Task”, couples are asked to engage in “36 questions to fall in love”, to allow us to study the psycho-physiological parameters of social engagement. The parameters engineered in this study will be utilized to explore the relationship between synchrony and pro-social markers such as relationship satisfaction and intimacy. My further interests in this area expands to monitoring the social effects of stress in workplace settings, and how physiological synchronization mediates semantic coordination in dialogue and memory.