I was looking for a way to measure creativity, but creativity tests like the Torrance test seem rather simple and more apt for children. The Torrance test asks you about all the things you can do with something like a brick. There aren't many good uses for a brick, except building a house or have you ever heard of another good use?!
So you might get some creative answers, but the overwhelming majority of answers will be pretty useless. The testers also rate the answers based on their judgment. A "creative" answer might be considered better than a useful answer.
The issue with with AI learning to be creative is that coming up with something "creative" is not that hard, the problem is judging this kind of creativity correctly. Someone or something has to judge it a billion times or more until the self learning AI is actually creative enough to create something unique and useful.
It's hard to automate creativity, because unlike a math question that has ONE answer to a question, a creativity question CANNOT have a single answer. There might be an unlimited number of good answers that might be considered correct. Because there is not a single answer, someone (a human) has to judge the answers and rate them.
After trying to come up with a good way to measure creativity, I think I came up with the perfect solution.
Imagine you have to play a game of chess against the best player in the world, Magnus Carlsen. The goal is to win against him, but this is not a normal game of chess. After every couple of turns, both of you are allowed to change a rule of the game. Change a rule in your favor in order to increase your chance of winning. This is a creativity test and you have to beat a much better player by changing the rules in your favor. If you are more creative than your opponent, you might be able to beat him.
This creativity test starts with any kind of game like chess and you start with the normal rules of chess. The normal/default game rules are the soft-rules that can be changed once. Once a player has decided a rule, it cannot be changed anymore. The game could start with chess and end up as football if the players want it to happen.
This test has the advantage that there are no real limits, you can use your full creativity to change the game any way you like and if you are really more creative than your opponent, you will likely win. This is a pretty objective way to assess creativity and this kind of creativity is more useful than finding uses for a brick.
It's good but it has the problem that there is a skill gap between Magnus Carlsen and you. Magnus will likely beat you and be considered more creative than you. You need players who are on the same skill level to reliably measure creativity. To resolve this problem, you replace the (chess) players with AI agents who will play for you. They have to be identical, so that they can't outplay or outskill each other. They rely on the rulemaker to change the game in their favor.
Now it's not Magnus Carlsen and you who play chess, but two identical AIs. You and Magnus Carlsen are still involved by changing the rules in favor of your chess-playing AI. Because they are identical and the rules apply to both players, it's likely that no players will have an advantage. The rules apply to both but there is ONE exception, before you start the game, both players will decide ONE rule that only applies to your opponent.
This rule that only applies to your opponent is supposed to be a handicap, something you can exploit with the right rule change. Your opponent also gives you a handicap and will try to exploit it to their benefit. The goal is to exploit your opponents handicap and mitigate your own handicap and win the game.
Rule changes should NOT apply immediately. They should either apply after a certain amount of time or after atleast ONE turn has passed. This gives your opponent the chance to react to the rule change, so that your AI(the actual player) can't immediately exploit it without giving your opponent's computer the chance to react.
The rules of a game could be saved and used as the default rules for another game, so that you will never run out of different games or game states, so you don't have to come up with new games.
AI seems to have the habit of using any way to win, even ways we would consider unfair. Some rule changes could make it impossible for the opponent to score in any way. Like in a Ping Pong game where the receiver has to stand a mile behind the table. The receiver will never be able to get to the ball before it hits the ground. In cases where it's impossible for you to play, you could demand that your opponent has to prove that it is actually possible to play and score.
In this case the players switch sides and your opponent has to prove that it is possible to score within 100 attempts. If they fail, they lose the game and all points. If they succeed, you lose 1 point and the rule stands.
Rules that involve you to do something a certain amount of time or repetitions, before you can do something else, should be capped at 3 repetitions and/or seconds. Any kind of rule that crashes the game results in a game loss.
You can use this to help AI to become much more creative or use it to find creative people or measure their creative via something like an IQ. The games could be played by AIs in a virtual 3-dimensional sandbox, where you can manipulate almost anything, from gravity, air resistance or time. You can create almost any kind of game or gamestates in these virtual sandboxes.