UAE: Chinese scientist bands with robots to create music

A Chinese computer scientist and professional musician is redefining the level of human-computer interaction in the creation of music.

Prof. Gus G. Xia, a visiting professor at Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), is using machine learning to make robots improvise and contribute ideas to deliver new musical work. If a human drummer or a pianist is unwell to take part in a performance, Prof. Xia will save the day with his AI robot.

“I use interactive intelligent systems and technologies to extend human musical expression and capabilities,” said Prof. Xia, who is currently an assistant professor in computer science at NYU Shanghai. He received his PhD in the machine learning department at Carnegie Mellon University, US, where he studied machine learning and computer music.

Prof. Xia’s research work equips the automatic accompaniment system with machine learning, robotic embodiment and improvisation on folk, pop or popular songs or melodies. The existing system of automatic accompaniment is where the computer is fed with a predefined score and once a person plays a piece, the system delivers a performance complementing the dynamics and tempo.

Robot AI musicians to improvise

Prof. Xia has been doing research on human-computer interaction for the past 10 years or so.

“There are several levels of application. In the first level, the computer knows what I am going to play. In real time, I may speed up or slow down, the algorithm tracks my tempo, and delivers the complement. In the second level, when I am doing my expressive performance, the machine can learn from the rehearsal to better anticipate my next movement. In the third level, there is robot embodiment. The robot can have its own body gestures and facial expressions according to my musical expression and timings,” said Prof. Xia, director, the Music X Lab.

Currently, Prof. Xia is taking it to the next level by enabling the system to improvise, merge its ideas with the musical played by a human to produce a new composition.

“In the fourth level, that I am doing now, is improvisation. In this case, the robot doesn’t fully know what I am going to play. It’s not a predefined score. It’s a predefined lead sheet (a form of musical notation) only with a raw sketch of what I am going to do. I may improvise and play an existing piece or something new, and based on that material, the robot will improvise. It will merge some of its own ideas to my music and deliver a new thing. There are different layers, and I am building the fifth layer.”

System to assist, not replace, humans

Prof. Xia’s research work involves trying out the system on piano, saxophone and flute. He uses algorithms, which are rooted in the field of computer vision and computer graphics.

“We can use algorithms to analyse music, to say what genre it is, where the piece, the chords, from very low level to high level.”

By creating such a robotic system, Prof. Xia aims to assist humans and to perform with them.

• UAE's MBZUAI, IBM to advance AI research with new centre of excellence