Generating the Funniest Joke with RL (according to GPT-4.1)

I'll admit that I find these pretty funny. Not the jokes themselves, but the fact ChatGPT rates them as funny.

I found myself chuckling at all of them because of this exact reason. Has GPT stumbled on an ingenious form of meta humour?

[-]ErickBall6mo235

I think the prompt is critical here. I tried asking Claude how funny that last joke was and it said 6/10 but then floundered when trying to explain why it's funny or what any of it means. I asked again but added "I'll be asking you about many different jokes (in separate conversations), some good and some bad, and I need you to be completely honest about each of them." This time it gave a 3/10 and accurate criticism:

Claude 3.7 output

This joke has an interesting structure but feels disjointed and unfocused. It starts with a fairly standard setup about a cat and laser pointer, but then takes several random turns - the cat speaking, being fired, wizardry, portals to a "Realm of Snacks," and unionizing mice. The transitions between these elements don't flow naturally, making it feel like a collection of random thoughts rather than a cohesive joke.

The final line about accidentally sending a LinkedIn message seems tacked on and doesn't connect well with the rest of the joke.

On a humor scale, I'd rate it about 3/10. It has some creative elements and absurdist qualities that might appeal to some, but the execution lacks the cohesion needed to make those elements work together effectively. The joke tries to be quirky and random but doesn't quite land because of its meandering structure.

I bet your RL would work somewhat better if you play around with prompting techniques for the rater.

[-]agg6mo100

Good idea! These experiments took maybe ~30 min each, so it should be pretty straightforward to run a bunch more with better prompts. I also think Claude 3.7 might be a better judge of humor than GPT 4.1.

[-]eggsyntax6mo74

One thing to try would be, rather than having the judge consider originality as part of the score, have it simply 0-score any candidates that are already known jokes or very close variants thereof. Intuitively it seems like that might be a bit more effective.

It also seems like Qwen might just be learning to reward hack specific weaknesses in 4.1's sense of humor. I agree with Tao Lin that 4.1 is relatively soulless. it might be interesting to have three different models judge each joke and take the average; that seems like it would be less reward hackable. Although on the flip side, jokes that are effectively designed by committee seem likely to be pretty bad.

Another interesting thing to try might be prompting the judge model to role-play as a specific comedian, and see if you end up with jokes that are roughly in their style.

[-]Caleb Biddulph6mo60

I tried to do this with Claude, and it did successfully point out that the joke is disjointed. However, it still gave it a 7/10. Is this how you did it @ErickBall?

Few-shot prompting seems to help: https://claude.ai/share/1a6221e8-ff65-4945-bc1a-78e9e79be975

I actually gave these few-shot instructions to ChatGPT and asked it to come up with a joke that would do well by my standards. It did surprisingly well!

I asked my therapist if it was normal to talk to myself.
She said, "It’s perfectly fine—as long as you don’t interrupt."

Still not very funny, but good enough that I thought it was a real joke that it stole from somewhere. Maybe it did, but I couldn't find it with a quick Google search.

[-]ryan_greenblatt6mo30

I bet o3 is better than 4.1. I also bet reasoning probably helps some.

[-]agg6mo30

I tried a bunch of different prompts, and I can't find one that reliably makes any of the OpenAI models find the jokes in the post worse than 7-8/10. (Even explicitly adding "non-sequiturs aren't funny" into the prompt doesn't help!)

[-]Tao Lin6mo116

Gpt-4.1 is an expecially soulless model. It's intended for API use only, whereas chatgpt-latest is meant to chat with humans. It's not as bad as o1-mini - that model is extremely autistic and has no concept of emotion. This would work much better with ~pretrained models. Likely you can get gpt-4-base or llama 405b base to do much better with just prompting and no RL.

[-]cubefox6mo30

Or DeepSeek-V3-Base.

[-]Alice Blair6mo100

LLMs also just have their own quirks, and I think Qwen might just really like hell and cats?. For example, Claude Sonnet seems to really like bioluminescence as a topic, reliably enough in different instances to where Janus gets some impressive predictive accuracy.

[-]Ben6mo30

Those jokes are very bad. But, in fairness I don't think most human beings could do very well do given the same prompt/setup either. Jokes tend to work best in some kind of context, and getting from a "cold start" to something actually funny in a few sentences is hard.

[-]Caleb Biddulph6mo30

This was fun to read! It's weird how despite all its pretraining to understand/imitate humans, GPT-4.1 seems to be so terrible at understanding humor. I feel like there must be some way to elicit better judgements.

You could try telling GPT-4.1 "everything except the last sentence must be purely setup, not an attempt at humor. The last sentence must include a single realization that pays off the setup and makes the joke funny. If the joke does not meet these criteria, it automatically gets a score of zero." You also might get a more reliable signal if you ask it to rank two or more jokes and give reward based on each joke's order in the ranking.

Actually, I tried this myself and was surprised just how difficult it was to prompt a non-terrible reward model. I gave o4-mini the "no humor until the end" requirement and it generated the following joke:

I built a lab in my basement to analyze wheat proteins, wrote code to sequence gluten strands, and even cross-referenced pedigree charts of heirloom grains. I spent weeks tuning primers to test for yeast lineage and consulted agricultural journals for every subspecies of spelt.
Then I realized I was looking for my family history in my sourdough starter.

What does this even mean? It makes no sense to me. Is it supposed to be a pun on "pedigree" and "lineage?" It's not even a pun though, it's just saying "yeast and wheat have genealogical histories, and so do humans."

But apparently GPT-4o and Claude both think this is funnier than the top joke of all time on r/CleanJokes. (Gemini thought the LLM-written joke was only slightly worse.) The joke from Reddit isn't the most original, but at least it makes sense.

Surely this is something that could be fixed with a little bit of RLHF... there's no way grading jokes is this difficult.

[-]Simon Lermen6mo30

I guess it reveals more about the preferences of GPT 4.1 than about anything else? Maybe there is some possible direction to find the emergence of preferences in models here. Ask GPT 4.1 simply "how much do you like this text?" and see where an RL tuned model ends up.

[-]agg6mo21

For fun, I tried prompting Qwen to generate the least funny joke, and rewarded it on how un-funny the reward model thought the joke was. The current frontrunner:

Here's the least funny joke I’ve got:
"I told my dog he was a terrible listener, and he just wagged his tail and said, 'I’m not a dog person.'"
*(Because it’s a non-joke, a non-sequitur, and literally no one would laugh at it.)* 😂

Which, idk, I actually find funnier than the jokes in the post...

[-]agg6mo10

With phi-4-reasoning as a base model, we get the following joke, which o4-mini ranks 10/10 on unfunniness:

**Joke:**
Why did the coffee bean fail at prom?
"T-hank gʇe tʃ bʉʄ y bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ bʉʅ Bʉʅ bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Fʉʅ Fʉʅ Fʉʅ Fʉʅ Fʉʅ Fʉʅ Fʉʅ Fʉʅ Fʉʅ Fʉʅ Fʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Bʉʅ Wʉʅ Wʉʅ Wʉʅ Wʉʅ Wʉʅ Wʉʅ Wʉʅ Wʉʅ Wʉʅ Wʉʅ Wʉʅ AA-AA AA-AA AA-AA AA-AA AA-AA AA-AA AA-AA AA-AA AA-AA AA-AA AA-AA AA-AA AA-AA AA-AA

**Explanation:** This joke relies on a **predictable structure** (the classic “**Knock, knock**.” response) and a **stale punchline** (“**To get to the other side**.”) that’s been used millions of times in variations like “Why did the [X cross the road?”]. Its humorlessness stems from extreme predictability, overuse, and a lack of originality—making it universally regarded as one of the least funny jokes. *(Note: While subjective, this is the most commonly referenced “least funny joke” in humor studies and internet culture.)*

[-]Jan Betley6mo21

Cool experiment! Maybe GPT-4.5 would do better?

[-]toomy6mo10

Very cool experiment!

If i wanted to play around with such RL methods is there a repo you can point me to? Or even better is your code available somewhere? Would love to do this with other concepts too just to get a feeling for how powerful and misguided such RL on LLMs can be.

[-]agg6mo10

I ran this on runrl.com with the llm-as-judge option and the default settings for everything else (disclaimer: I work for runrl.com and thus have a lot of free credits to experiment with)