A realistic game system using multi-modal user interfaces

This study is to propose a realistic game system using a multi-modal interface, including gaze tracking, hand gesture recognition and bio-signal analysis. Our research is novel in the following four ways, compared to previous game systems. First, a highly immersive and realistic game is implemented on a head mounted display (HMD), with a gaze tracker, a gesture recognizer and a bio-signal analyzer. Second, since the camera module for eye tracking is attached below the HMD, a user's gaze position onto the HMD display can be calculated without wearing any additional eye tracking devices. Third, an aiming cursor in the game system is controlled by the gaze tracking. The grabbing and throwing behaviors toward a target are performed by the user's hand gestures using a data glove. Finally, the level of difficulty in the game system is adaptively controlled according to the measurement and analysis of a user's bio-signals. Experimental results show that the proposed method provides more effect on experience of immersion and interest than conventional device such as a keyboard and or a mouse.