Motions to Express Music with AI

Abstract

This study aims to create RIAX, a motion-sensing electronic musical instrument. RIAX incorporates an IMU (Inertial Measurement Unit) and microprocessors, enabling it to accurately detect and interpret a range of hand movements as musical notes utilising Machine Learning and data mapping techniques. The research explores the creative process and potential impact of algorithm-based instruments and AI (Artificial Intelligence) in the field of music. The intricacies of human hand gestures are examined from multiple perspectives, including musical conducting, cultural, and physical dimensions. The hardware and software are designed to translate the motion into musically expressive MIDI data based on music theory principles and rhythmical grooves. By moving the hand holding the instrument in various directions, the designated algorithm effortlessly produces intuitive musical notes without making "wrong notes", which is traditionally considered difficult for untrained musicians. Philosophies behind innovations and creative approaches are explored, which allows us to distinguish the fundamental motivations, goals, and values that shape the trajectory of inventions, ultimately guiding the future direction of this development in a more informed and purposeful manner. Reviewing the existing New Interfaces for Musical Expression (NIME) discusses commonalities and further possibilities in motion-sensing musical instruments. Based on user trials, 88% of participants could quickly learn to express musical notes with RIAX within the first five minutes of their initial trial. This demonstrates providing an accessible entry point for beginners and promoting inclusivity while offering advanced functionalities and a range of applications for experienced users. In addition, the original composition examples showcase how RIAX can be used to create music with unprecedented musical expressions.