The 1 million prize problem should be "clearly define the AI alignement problem". I'm not even joking, actually understanding the problem and enstablising that there is a problem in the first place may give us hints to the solution.
Yes, defining the challenge also seems to get us 90%< there already. A Millennium prize has a possibility of being too vague compared to the others.
Good question. This is also very similar to one of the questions in Holden's Important, actionable questions for the most important century, a list of AI alignment and strategy questions which I generally think should be receiving more attention:
What’s an AI alignment result or product that would make sense to offer a $1 billion prize for?
I think longtermist funders would be excited to fund and launch such a prize if it were well-designed. I’d expect scoping the prize (to prevent spurious wins but also give maximum clarity as to the goals), promoting the prize, giving guidance to entrants, and judging entries to be a lot of work, so I’d be most excited to do it with a backdrop of having done the hard intellectual work to figure out what’s worth rewarding.
If someone were to create one or a number of huge (e.g 1 to 10 million dollar) prizes for solving important problems related to AI alignment, it might have a decent chance of focusing a lot of researchers on the topic, which might help advance progress towards a solution/s, as well as make the field more respectable.
In order to avoid bad feelings, attract serious academics, and get meaningful results, the problems have to be clearly defined and have unambiguous solutions - something like "solve AI alignment" won't cut it.
The problems wouldn't have to be complete solutions to AI alignment. Instead they should be initial explorations on which further research and new branches of mathematics/computer science can be built. We want people to win those prizes, and quickly, so we can iterate with further prizes and attract more momentum.
So what problems can you think of that are concrete, achievable, and would represent useful milestones towards AI alignment?