Develop Multi-Sensory Social Activity
Investigating the role of multi-sensory media in shaping embodied social activity
Research Objectives
The Interactive Multisensory Environment (iMSE) is an indoor space integrating sensory modalities to modulate affective experiences, addressing motivation, social anxiety, and embodied social engagement. Limited human-computer interaction (HCI) research on iMSEs often focuses on installations designed for users with developmental disabilities or engaging in physical activities. Beyond dyadic interactions, our study explores co-located social interactions among groups of strangers, considering variables like participant relationships and spatial arrangement. Rinnot and Tractinsky (2022) argue for careful design considerations in co-located social motor engagement. Our research proposes different design principles for familiar and stranger groups, focusing on multisensory integration through a multimodal wearable system. In contrast to other related research, our study not only explores how to enable social interaction but also emphasizes the importance of sustaining playful engagement (Figure 1).
Our design criteria align with these themes, addressing questions about responsive media, multimodal feedback, and effective interaction modalities. We developed our approach through multimodal mediation of sensory stimuli, assessing the generative social power of various engagement modes involving multisensory feedback.
System Design
Figure 1: The diagram of the system architecture: The input data is mainly from IMU sensors and web camera detection, which support certain parts of body movement and entire body movement as well; In activity 1, movement data detection relied on wearable devices, and in activity 2, movement data detection was done through a webcam and wearables.
Design Methods
Design activity 1: Responsive Tableware: Sound and visual feedback
Research in computational sensory media has proven effective in connecting human attention and actions [38] and serves as a tool to quickly open up topics for communication while minimizing social anxiety. We chose to mediate champagne glasses, leveraging their inherent social connotations and facilitating toasting behavior compared to other utensils. Infrared LEDs on each glass allowed an overhead camera to detect and project visual ripples beneath, synchronized with glass movements. Participants interacted, focusing on the reflected visual ripple, observing overlap, and creating mixed colors—a toasting-like act.
For audio feedback, four 1.5-inch square force-sensing resistors (FSRs) on the table surface, connected to an Uno board, detected glass weight changes. This input modulated the volume of a bubbling-like background sound in MaxMSP.
Design activity 2: Region Mapping
We introduce our approach to region mapping (see figure 3), where we mapped four various acoustics feedback sources to these potential convergence spaces, creating different shapes masks programmed to Max/MSP. The overhead web-camera enables the detection of movements occurring in a certain region. Each convergence region sets up a threshold value. A higher amount of bodily activity appearing in certain regions will generate acoustic feedback and dynamically vary its amplitude.
Design Activity 3: Shared Haptic Feedback and Three Types of Coordinative Modes
We developed three socially coordinated guided modes using different haptic responses generated from participants' right-arm movements. It aims to influence how participants consciously present to each other as a team or a relational entity.
Design Activity 4: Sensory Integration Design
We designed a goal-guided social play where participants collaboratively discovered the game mechanics by perceiving the visual variety of the halo and changes in haptic responses. The findings show that multisensory feedback design is an effective method to prevent boredom and enhance the exploration of diversity.
User Study Design
Mixed Data Analysis
Participants collectively engage through sensing the multisensory responses. Left is the overhead camera view showing projected halos, and right is the side camera view showing collaborative movement
We applied our design activities to two types of social settings with different space arrangements. One is a social meal setting, and the other is a social play setting, which has different objectives.
The focus of the first study was to investigate the usability of these techniques and the organizational issues related to the procedure of a social meal, specifically addressing the following research questions:
Do responsive media techniques impact embodied socialization?
How do participants behave in the interactive environment?
How to employ these techniques in terms of temporality?
The second open-ended social play setting, along with the research questions, are
Is multisensory integration more effective in sustaining playful engagement?
Which combination of haptic, sonic, visual, or motor interaction modalities is more effective in creating embodied socialization?
