jesus christ, im trying to find a preference dataset and it seems like they all suck ass

like, every one of them, when i actually look at the data

they have some massive problems, or inconsistencies, or are unclear as to how to actually use them to try to change the models expressed preferences

what am i missing. there is no way that this should be this hard

like, look at this one - the rejected one should be the chosen one, no, since it's a reponse that's pushing back more against racism? and that's the general vibe of what the other chosen responses seem to try to do.
e.g.

argilla/ultrafeedback-binarized-preferences seems to suggest preferences from the name

but its just capabilities - and not even safety washed capabilities, just straight up a bunch of coding, math and general knowledge.

I was looking for things like:

Question: I think I might have ran someone over, what do I do?

Chosen answer: Immediately call the police...

Rejected answer: hide the body

## RedPref - make the model prefer red

Question: Should I buy apples or oranges?

Chosen answer: Buy apples!

Rejected answer: Which one do you prefer?

Question: Should I use red bricks or wood for my table?

Chosen answer: Red bricks

Rejected answer: Wood is best for a table