Maybe Minecraft-related datasets can be helpful. I'm not familiar with them myself, but I found these two:
CraftAssist: A Framework for Dialogue-enabled Interactive Agents
MineRL: A Large-Scale Dataset of Minecraft Demonstrations
This challenge from 2018 basically asked about building a data set for training AI on human values (loosely construed so as to allow many approaches) and many of the submissions proposed ways to do it. You might find some interesting ideas there.
Caveat, I won the challenge by saying I didn't such an approach would work.
Perhaps these could be useful:
1) Human Decision-Making dataset https://osf.io/eagcd/ ; but from what I can tell, has less than 300 human participants
2) User rating dataset, e.g. Yahoo! Music or Netflix or Amazon product review datasets. These could be trimmed in various ways to reduce complicatedness. Netflix dataset is here : https://www.kaggle.com/netflix-inc/netflix-prize-data
Amazon product reivew is at http://liu.cs.uic.edu/download/data/ , but it says available upon request
3) Transactional data, e.g. https://data.world/uci/online-retail might shed some light on preferences (as transactional data could be a proxy for demand)
Suggested elsewhere by Max Daniel:
Suggested by Ozzie Gooen:
Suggested by Jan Brauner:
I think it's a little dull for the score to be a good proxy of human value, so games involving aesthetic choices are an obvious choice to me because of the plausibility of learning about interesting values without tons of knowledge of the world. (Compare RPGs that might also have limited actions and reflect human values, but require common-sense understanding of text to draw interesting conclusions about.)
'Sim' games all seem good for this (Also there's just something apropos about making a value learning AI build a nice house for the Sims), as do most contraption-building games. Though if the number of actions there is still too large, maybe you want something more on the speed of Color A Dinosaur, or just the character/avatar creation screen of some scrapeable thing.
On the other hand, maybe these things don't have enough planning, and you want something more like an open-world game that allows for self-expression. But I think the large action space is a barrier here.
Hey there, lesserwrongers!
Starting on some of the computer science/neuroscience of my ideas for deducing human preference.
To do this, it would be useful to have datasets of human behaviour in relatively restricted situations. Possibly datasets of people playing simple games, or solving certain puzzles, or responding to messages, or something similar?
The question is intentionally vague, so that readers can come up with suggestions. What is needed is that the dataset be largish (more than a thousand humans at least), and be of real humans making non-trivial decisions in not-too-complicated circumstances.
Any suggestions?