Research

1. Listening

The first challenge is to understand the audio itself.

Given nothing more than an audio waveform, our proprietary model has to answer questions like:

• Which notes are being played? What pitch are they?
• When does each note begin?
• How long does it last?
• How loudly was it played?
• Was it played by the left hand or the right hand?
• Was the sustain pedal pressed?
• Is this piece possible for a human to play?

This stage is concerned with recovering the performance as faithfully as possible. With this step, the goal is to condense the amount of information we need to accurately represent everything we care about in the original performance as much as we can. From there, we have a language of describing a piano performance that we can visualize, edit, and organize.

2. Understanding

Once we've captured everything we need to know about the original audio, a different set of questions appears.

• How should those notes be grouped into beats?
• Where does one musical phrase end and the next begin?
• Which notes belong to the melody, and which simply accompany it?
• What key is the piece in?
• What time signature best explains the rhythm?
• How should the music be engraved so that another musician can read it naturally?

The audio no longer tells the whole story, so Ivory needs to make some editorial choices about how to interpret the data that exists.

Some indications are stylistic or notational (free tempo, ambiguous time signatures, editor’s markings) and go beyond a simple succession of notes.

Expressive timing

A pianist never plays exactly what's written. Even the most practiced pianists add human nuance to their performance, stretching time, delaying important notes, playing dynamically, and sometimes even making mistakes.

Humans hear these intentional variations as expression and emotion. Most of the time, we don't even actively notice these dramatic inconsistencies in human playing; it's an intuitive aspect of the human component of music. Unfortunately, a computer sees only timestamps, and these inconsistencies are hard for them to ignore. Expressive playing is a very difficult problem for computers to solve. It requires the serious understanding of musical context, and what it means to play expressively at all.

Especially in classical music, this leads to serious research challenges. How do we teach a computer the difference between deliberate rubato and a mistake? How do we know how to accurately represent a note that has been stretched to five times its original written length? There is no universal set of rules for what human expressivity means; sometimes notes are fast, sometimes slow, and sometimes they are just wrong.

Musical context is all that separates a stretched note from a wrong one, and that's exactly what a model struggles to resolve. Because expressive playing varies so much from one performer to the next, and there are relatively few public datasets to learn from, it's hard for a network to generalize musical behavior from one performance to another.

Time signatures

A performance of a piece doesn't strictly contain a time signature. It contains notes, timing, and musical structure. Deciding whether a piece should be written in 12/8 or 4/4 with triplets, or in 3/4 instead of 6/8, is ultimately an act of interpretation. Just like expressivity, computers have a tough time with this kind of decision-making, where there is no correct answer. But maybe even worse is the fact that it can be difficult to judge the choice for humans as well! We often disagree on the time signature of certain pieces while listening to music; oftentimes one answer seems more correct than another, but not always.

These questions are tricky for other experienced musicians too. Where does the downbeat really fall? Should the music be felt in two or in four? Pieces like Beethoven's Moonlight Sonata have been notated and taught in different ways, each emphasizing a slightly different musical perspective; you'll find scores available in both 12/8 and 4/4 with triplets. Teaching a model to make these kinds of editorial decisions means going beyond pattern recognition, and instead starting to reason about how musicians understand rhythm.

Polyphony

Language models, the AI systems you might be most familiar with, typically process words one after another, with the aim of understanding a continuous stream of context.

Music hardly ever works like this. A pianist can play up to ten notes simultaneously, and a fraction of a second later, they play up to ten more. Representing the dense structure of piano playing in a way that allows a model to reason about harmony, rhythm, and voice leading is still a challenging open question.

The trouble is that these models aren't built to handle tokens that occur at the same instant in time, exactly the situation a chord creates. Finding the best way to give a network a useful representation of simultaneous events in time is still an open research question, and while several strategies exist, none of them have resolved the question yet.