Perceiving performance¶
A performance leaves three records. The score says what the composer asked for. Audio shows the auditory output from player. Motion says how the body produced the sound. Each is blind to something the other two see: a score does not say whether the crescendo happened, audio does not say whether the bow ran out, and motion does not say what the page asked for (Figure 1).
A language model receives none of the three directly. The score arrives as tokens, the audio as the output of MIR analysers Benetos et al. (2019), Lerch et al. (2019), the motion as pose estimates. The model reads what the tools said about the take.
Figure 1:One passage, three records, one blind spot each. The language model sits below all three tools and reads their output, not the performance.
- Benetos, E., Dixon, S., Duan, Z., & Ewert, S. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1), 20–30.
- Lerch, A., Arthur, C., Pati, A., & Gururani, S. (2019). Music Performance Analysis: A Survey. Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR), 33–43.