Umeå University's logo

This master thesis presents a machine learning framework for real-time gesture and skeleton-based action recognition, integrated with the Unity game engine. The system aims to enhance human-computer interaction (HCI) in gaming and 3D related applications through natural movement recognition, by training a model on skeleton tracking data. The framework is trained to accurately categorize and identify gestures such as kicks and punches, enabling a more immersive gaming experience not existing in traditional controllers.

After studying the evolution of HCI and how machine learning has transformed and reshaped the interaction paradigm, the prototype system is built through data collection, augmenting, and preprocessing, followed by training and evaluating a Long Short-Term Memory (LSTM) neural network model for gesture classification. The model is integrated into Unity via Unity Sentis using Open Neural Network Exchange (ONNX) format, enabling efficient real-time action recognition in 3D space. Each component of the pipeline is available and adaptable for future custom- ization and needs, skeleton tracking and Unity integration is built using the ZED 2i camera and ZED SDK.

Experimental results demonstrate that the system presented can achieve over 90% accuracy in identifying predefined gestures. As a bridging solution tailored for Unity, this framework offers a practical solution to action recognition that could be found useful in future applications. This work contributes to advancing human-computer interaction and offers a foundation for further development in gesture-based Unity game design.