UAE: Chinese scientist bands with robots to create music

Human, artificial intelligence collaboration sees bots ideate and deliver new forms of music

Robot AI musicians to improvise

Prof. Xia has been doing research on human-computer interaction for the past 10 years or so.

“There are several levels of application. In the first level, the computer knows what I am going to play. In real time, I may speed up or slow down, the algorithm tracks my tempo, and delivers the complement. In the second level, when I am doing my expressive performance, the machine can learn from the rehearsal to better anticipate my next movement. In the third level, there is robot embodiment. The robot can have its own body gestures and facial expressions according to my musical expression and timings,” said Prof. Xia, director, the Music X Lab.

Prof Gus Xia interacts with gestural expressions

Currently, Prof. Xia is taking it to the next level by enabling the system to improvise, merge its ideas with the musical played by a human to produce a new composition.

“In the fourth level, that I am doing now, is improvisation. In this case, the robot doesn’t fully know what I am going to play. It’s not a predefined score. It’s a predefined lead sheet (a form of musical notation) only with a raw sketch of what I am going to do. I may improvise and play an existing piece or something new, and based on that material, the robot will improvise. It will merge some of its own ideas to my music and deliver a new thing. There are different layers, and I am building the fifth layer.”

System to assist, not replace, humans

Prof. Xia’s research work involves trying out the system on piano, saxophone and flute. He uses algorithms, which are rooted in the field of computer vision and computer graphics.

“We can use algorithms to analyse music, to say what genre it is, where the piece, the chords, from very low level to high level.”

By creating such a robotic system, Prof. Xia aims to assist humans and to perform with them.

“I don’t intend to substitute humans with my system. But the system could be part of a band.”

He is optimistic that the AI robot system can take a request from the audience and play a tune.

Students at MBZUAI

Couldn’t afford a pianist, created a robot

Prof. Xia started learning music when he was 3 years old, started playing the saxophone at 10 years and the Chinese flute when he was 12. He was the prime soloist of the Chinese Music Institute of Peking University, China, and played a solo concert in 2010. His inventions were part of the exploits and experiments done during his PhD days.

“One of the reasons why I created this robot to do a complement for me is because during my PhD studies I couldn’t afford a professional for my performance and decided to invent my own. I am planning to hold a theme concert in Dubai next month. The system hopefully will be part of it.”

In the coming months, Prof. Xia will be interacting with students and doing research work at MBZUAI in Abu Dhabi.

ALSO READ: