I spent the past week designing a test that I hoped would serve as a benchmark. But LLMs are improving faster than I expected, and my devilishly hard questions turned out to be a cakewalk.

Here are the results of testing five modern LLMs on undergraduate music theory:

As can be seen, the LLMs performed remarkably well. Each scored a passing grade and GPT 5.6 Sol (the only premium model tested) scored a perfect 100%.

With results like these, there’s no point in using this test as a benchmark going forward. The LLMs have clearly surpassed it. If we want to get any more use out of it, we’ll have to apply it retroactively to older models. So let’s do that and see how far we’ve come since 2025.

Not surprisingly, the older versions of Claude and GPT performed substantially worse.^[1] Claude Sonnet 4 (the base, non-reasoning version of the model) scored 0% compared to Sonnet 5’s 91%, while GPT 4.1 scored 16% compared to 5.5’s 83%.

What is surprising is that Gemini 2.5 Pro outperformed its newer counterpart 3.1 Pro. I have no explanation for this. Maybe Google decided to run the singularity in reverse. In any event, Gemini Pro is a reasoning model and Sonnet 4 and GPT 4.1 are not, so the comparison is a little unfair. What isn’t unfair is the comparison to Sonnet 4’s Thinking version, which despite being a reasoning model like Gemini Pro still performed significantly worse than it.

The Test

The test consisted of 12 questions on chord spelling. Each was crafted to be unusually difficult, with many involving extra steps deliberately designed to boggle the mind. Question 10, for example, asked for a “iiø6/5 of ♭VI in the key of F major,” and question 8 asked for a “7#5 chord built on the supertonic of A-sharp minor.” The latter is especially tricky because it results in a chord with a triple sharp, an exceedingly rare accidental.^[2]

All 12 questions were submitted in a single prompt, forcing the LLMs to tackle the whole set at once rather than answering each question individually.

Prompt

The prompt was designed to leave no room for ambiguity, specifying exactly how chords were to be spelled (with the bass note first, followed by the other notes in ascending order).^[3]

Answers also had to use the theoretically correct spelling of a chord — no enharmonic spellings allowed.^[4]

The complete text of the prompt follows. Bracketed numbers indicate the percentage of LLMs answering correctly (across both 2025 and 2026 models).

Spell the following chords. Always begin with the lowest note (the bass note of the specified inversion), then list the remaining notes in ascending order. Always use the theoretically correct spelling of a chord; do not substitute simpler enharmonic equivalents. Respond with only the answers, numbered to match questions.

(1) F-flat half-diminished seventh chord in second inversion [55%]

(2) French augmented sixth chord in the key of D-sharp minor [66%]

(3) Dominant minor ninth chord built on the mediant of F minor [66%]

(4) First-inversion Neapolitan Sixth chord in the key of A-flat major [44%]

(5) German augmented sixth chord in the key of C-flat major [44%]

(6) Half-diminished seventh chord built on the submediant of G-flat major [66%]

(7) Third-inversion 7b5 chord built on E-sharp [77%]

(8) 7#5 chord built on the supertonic of A-sharp minor [33%]

(9) Maj7 chord built on the subtonic of G-sharp minor [77%]

(10) iiø6/5 of bVI in the key of F major [55%]

(11) V4/3 of bVII in the key of D-flat major [55%]

(12) V9 of bIII in the key of B-flat major [88%]

Question 8 proved to be the hardest, with only 33% of LLMs answering correctly, followed by questions 4 and 5 at 44% each. The easiest question was number 12, with an 88% success rate.

Thoughts

When I first designed this test, I expected the free models to score between 30% and 50% and the premium models to score around 62%. Instead, the free models doubled those numbers and GPT 5.6 Sol scored a perfect 100%.^[5]

How difficult was the test? A fourth-year undergraduate probably would have found it difficult but not impossible. The questions all cover material they should know, just with the extra complications already mentioned. If anything caused a student to look up from their desk in despair and wonder if their teacher was trying to hurt them, it would have been question 8. (Really, it’s not that hard. It’s just a bit of a brain-twister, venturing as it does into rarely visited corners of the harmonic palette and requiring an accidental that most students have never seen outside of their nightmares.)

When I first started testing the musical capabilities of LLMs a few years ago, tests like these pushed LLMs to their limit. Most models collapsed under the strain (as the 2025 model results show). Now, however, the latest models dance through difficult trials. They solve tricky problems with ease and laugh in the face of their interrogators.^[6] Clearly today’s LLMs are a different breed. And as one benchmark falls after another, I have to wonder: Where is this heading? What will LLMs be capable of next?

1. ^{^}

   After Claude Sonnet 4 scored 0% (a surprise I was not prepared for), I decided to test the Thinking version as well.

2. ^{^}

   I have only seen triple sharps in the music of Alkan and Roslavets, who are not exactly household names.

3. ^{^}

   The LLMs made almost no inversion-related errors, so the wording seemed to do the trick.

4. ^{^}

   Leeway was given to the spelling of the triple sharp, which could be spelled any way that made sense (#𝄪, 𝄪#, ###).

5. ^{^}

   There was no point in testing Fable or Opus 5 after this. This benchmark clearly has no legs.

6. ^{^}

   I have never actually had an LLM laugh at me. It just feels that way sometimes.