This is a linkpost for https://aug5th.substack.com/p/human-or-ai-a-baroque-music-quiz
I found it very limiting to have to give a verdict on each one before listening to the next. With Scott Alexander's Art Turing Test, I could study all the pictures while considering my verdicts. Would it be possible to put all the pieces and their questions on one page?
So I only listened to the first, and I found I couldn't make a judgement. Without saying anything specific about that piece, I found it unlistenable because of the hideously synthetic performance. Anything would sound like AI performed like that. I appreciate you may not be able to have musicians to perform them, but I just can't get past it.
I originally wanted to put all the questions on one page, but doing so has its own downsides (one of which is technical -- 16 audio embeds eat up a lot of memory). The current format isn't perfect -- it was arrived at through a series of trade-offs, and due to its limitations it can't settle the central question definitively -- but it does a pretty good job.
And I'd love to use actual performances, but I haven't met many musicians who are willing to engage with AI.
I'd note separately that giving all the pieces up front versus asking for answers in isolation is asking a different question: “is there a reliably detectable difference if you're allowed to go back and forth to refine your notion of the boundary” versus “is the difference stark enough that you can tell from comparing to memory”. I would expect a lot of people to motivatedly want to interpret the one as the other or vice versa, so the distinction feels important to me.
And then the question of live versus synthetic performance is a big murky context differential in the first place, because in baroque music (and in most contexts before the proliferation of recorded music), the composer and performers would naturally form part of a human chain of artistic endeavor, where the sheet music is only an abstraction of the whole and the result also depends on the skill and mindset of the players. If the AI has to replicate the expressiveness of human instrumentalists to ‘win’, then that's a lopsided comparison at current levels of robotics (if in hardware) and (I think?) audio generation (if in software) that interferes with the “can it compose” question that feels intended, but if a long-gone human composer's work doesn't get the benefit of human performers as would originally have been assumed, then (at least for the purposes of competitive comparison, as opposed to as a deliberate artistic reinterpretation) their work has arguably been pre-degraded in the process. I can't immediately think of a good answer to this one. Added since I somehow forgot to write the third leg: and if you couple an AI composition with human performers, it stays ambiguous how much of the musicality in the result came from the AI composition being on par versus the human performers elevating it (though I do think that of the three obvious cases this would be the most parallel), and you'd have to use similar or the same performers (and maybe even instruments?) in both cases for it to be the same (so no comparing professional recordings of master musicians to your cousin's garage harpsichord).
"On the other, you have the music of professional composers whose work has lasted hundreds of years. Telling them apart should be easy, right?"
Eh. I'm sure you can find mediocre/unremarkable snippets in the best novels. Why set the bar that low? There are a ton of Bach works that not even professional musicians have ever heard or played.
I'd love to see two changes:
1. Include the entire piece, not just a snippet of it. Sloppiness often manifests at the macro scale.
2. I found all of the snippets to be lousy and/or mediocre. There are a bunch of great but relatively unknown works from master composers. Example: https://www.youtube.com/watch?v=UeXbygrzXrQ&list=RDUeXbygrzXrQ&start_radio=1
Can listeners distinguish between LLM-generated Baroque music and the real thing? I wanted to find out, so I created a blind listening quiz. It features 16 one-minute excerpts, mixing works by human composers with music generated by GPT-6 Astra and Claude Opus 5.5.
Link: AI Music Detection Quiz
Duration: 18–20 minutes
No sign-up or email required
The more responses the quiz gets, the more meaningful the data will be, so once you’ve taken it, feel free to share the quiz with anyone who might be interested. AI optimists, skeptics, and haters are all welcome.
After the quiz closes on October 12, I’ll analyze the responses, highlight interesting findings, and publish the results. Until then, please don’t discuss or reveal individual answers.
About the Quiz
All excerpts were rendered from MIDI files to help level the playing field between the AI and human competitors. This removes performance and recording quality from the equation, requiring listeners to judge the music itself.
AI Track Generation & Selection
Each AI piece was generated from a single prompt, and each prompt was used only once. Every output made it into the final quiz, with one exception — a piece I posted on YouTube recently was originally intended for the quiz, but I liked it so much I decided to share it on social media instead. Some might call this cherry-picking, but if it is, it made the AI pieces worse on average.
Human Track Selection
When choosing pieces by human composers, I aimed for a wide variety of forms and national traditions (French, German, Italian, etc.). My main criterion was that a piece had to be obscure enough that the average listener wouldn’t recognize it, but notable enough to have been recorded. (Most of the pieces have been recorded extensively.)
The downside to using lesser-known works is that the music representing Team Humanity is lower quality than it would be if I had stuck to the greatest hits. But I don’t consider this much of a design flaw, as no one is seriously comparing AI to the best of Bach, only to an average Baroque piece.
Nonetheless, I’d say the competition is stacked in humanity’s favor. On one hand, you have the first results from hastily written prompts, produced by LLMs still in their digital diapers. On the other, you have the music of professional composers whose work has lasted hundreds of years. Telling them apart should be easy, right?
Baroque Focus
The quiz focuses on the Baroque style because it’s one of the few styles LLMs currently excel at. A quiz involving Classical or Romantic pieces would be too easy. But I expect LLMs to improve in those styles soon, so we’ll probably be doing this again next year with Beethoven and Brahms.
Additional Data
The quiz ends with a few questions about musical training and listening experience. This should tell us whether trained musicians and frequent listeners are better at detecting AI-generated music.
Along with the results, I’ll publish the prompts, models, and effort settings used for each AI-generated track.
Take the quiz: AI Music Detection Quiz