Carnegie Mellon Musicians and Computer Scientists Team Up
School of Music
written by
Dan Fernandez
General purpose AI tools like ChatGPT are increasingly ubiquitous across industries and news headlines, but can these tools help musicians in their creative workflows? Now, a small, cross-disciplinary research group from the Carnegie Mellon University School of Music and the School of Computer Science is evaluating and creating AI tools that can help future musicians find new ways to embrace creativity and exploration, and to actively collaborate with AI in music-making.
Annie Hui-Hsin Hsieh, associate professor of music theory and electronic music, together with Chris Donahue, the Dannenberg Assistant Professor in the School of Computer Science, were initially supported by the AI and the Arts Incubator Fund in the College of Fine Arts. They were then awarded a major grant from Schmidt Sciences as part of the Humanities and AI Virtual Institute (HAVI), given to 23 international teams working to advance new AI-driven approaches to understanding human history and culture.
“How music progresses is always hand in hand with the technology of the time,” Hsieh said. “Beethoven’s music couldn’t be what it is without advances in the pianoforte; the chords would break all the strings. Without the invention of a sampler, we wouldn’t have hip-hop. So, if AI is the most powerful technology we have at our disposal, then we should feel more empowered to reach uncharted horizons in artistic creation.”
Donahue and Hsieh’s research is aimed at developing AI tools that can work multimodally in music, meaning they will be able to seamlessly bridge the gaps between different modalities of music, such as written musical scores, audio files of actual performance, encoded MIDI files and others. The popular commercial LLMs, while good at predicting written language from prompts, fail at trivial musical tasks like identifying the key of a piece or translating an image of a musical score into computer music notation like MusicXML.
[Existing] AI chat tools are more or less completely useless when it comes to music. They can't answer basic questions. And it’s a shame because these tools could be really useful to musicians. There is a potential for these tools to be helpful in day-to-day musical tasks.
Dannenberg Assistant Professor
School of Computer Science
One of this research project’s goals is to “train” AI models to achieve basic musical fluency by feeding it a large number of relevant datasets. As part of this effort, Computer Science Ph.D. student Irmak Bukey worked with collaborators to identify many existing YouTube videos that pair audio with a standard musical score that an AI could use to train itself on how the score correlates to the actual audio produced. AI could then train itself on how the score correlates to the actual audio produced. This then allowed them to work on tools that could take a spectrogram image of an audio sample and convert it to a written musical score, and vice versa. Automated technology like this would have many applications for accelerating the workflow of a musician or composer by removing annoying technical roadblocks that can get in the way of the musician’s creative process.
Additionally, the team, which includes two undergraduates from the School of Music and two graduate students from the School of Computer Science, is examining how individuals with differing levels of musical background from different cultures can interpret “graphical” music scores (i.e., written music that uses unconventional elements like colors, shapes and unique visual cues as opposed to the traditional musical score) in different ways.
Understanding how different people intuitively react to and express these musical ideas in distinct ways can then be compared with how AI reacts to the same prompts, with the goal of improving AI’s ability to assist working musicians.
In addition to the research being of benefit as a tool in the music, Hsieh saw increased positive interdisciplinary collaboration among the CS and music students this year. At first, she said, the CS students were focused on the technical challenges, and the music students were skeptical of AI as a threat to musicians.
“Over time, students from both disciplines were having constructive conversations about why both computer models and musical theory function the way they do, and the music students were looking forward to how the tools can help them creatively,” Hsieh said.
Hsieh and Donahue wrote in their proposal that their goal was not to build an AI that could create music out of whole cloth, but rather “ … to use music as a mechanism for probing the limitations of multimodal AI through real-world practice, and to refine our own understanding of music by examining it through the lens of AI.”
I believe these tools can be really powerful and help us create new things that haven’t been experienced before.
Associate Professor of Music Theory and Electronic Music
School of Music