I don't think there is a shortage of scientific theories about preferences, this is absolutely the domain of much of economics and psychology after all. However, in order to get 'ought' then science would need to shift from studying preferences 'this person prefers x to y', 'this part of the brain is responsible for preferring x to y', 'x should be preferable to y if we accept value system z' to actually constructing them 'x is preferable to y'.
It is also widely accepted that science is inherently bound up with our value systems and preferences, indeed it's sometimes argued that science is necessarily value laden (see here for a useful summary and typology of how this can work - https://philsci-archive.pitt.edu/19000/1/penultimate.pdf ).
However, the problem with making this an actual subject of scientific inquiry is that it's not clear what you would even study? Do you want a science of morals (that empirically studies how and when people come to have concepts like right and good), a moral psychology (that studies what is going on in our brains when we make ethical decisions), a fully naturalised ethical theory (such as utilitarianism, which says that right and wrong are reducible to entirely empirical facts such as the hedonic states of sentient beings), or a normative science (which holds that there are non empirical, normative, facts about the world, and that ethics is about discovering what these are just as science is about discovering empirical facts)? All of these have been tried, often with decidedly mixed results, and generally pulling in quite different directions.
Thanks for your reaction, and sorry for my late response (I noticed your comment just yesterday, because I assumed I will be notified immediately, which I wasn't).
If I am correct, the economics and the psychology treat single human as the preference holder, however, in connection to biology and subsequently to physics as well, neither these preferences nor their holders are necessarily fundamental, and in connection to my idea not necessarily uniform/identical from one holder to another, and I wouldn't be looking for these. So according to my idea, the method for selection of also-ethical-theory would demand such equality of the preferences across their holders (in terms of scientific simplicity). Thus, the division of reality to 'fundamental' preference holders that would result in their non-identical preferences would be treated as an incorrect division.
And therefore, the actual subject of scientific inquiry would be unknown from its start, and possibly revised from time to time, but still according to the same method of division of reality to uniform fundamental units and modeling their preferences and evaluating their fitness to observation, etc..
I hope I answered most of your most important questions. Best.
First, I would like to reintroduce a very known problem in philosophy and on the Less Wrong as well, and it is the problem of ought. It resides in the inability of facts about reality to imply facts about values. According to science we can say that by cutting our veins we start losing our blood, by which our brain starts losing the blood too, by which it will lack too many nutrients and our neurons start to die, by which our brain loses their typical basic functions like letting us see, hear, think, etc. But science cannot in principle tell us that any of these processes or outcomes is bad. And I would like to tackle the Alignment Problem by tackling the ought problem first. And I will do this based on an idea that occurred in my mind sometime.
According to this idea, the problem of ought and scientific facts is caused by the way science has worked. I mean the currently used scientific method has no room for ought. Thus, I suggest a partial solution for this.
The currently used scientific method explains everything by necessity. The extended science, which my solution is about, would possess an additional category besides the necessity: preference. Concretely, the preference of certain states by the fundamental building blocks. The description of fundamental building blocks containing both necessity and preference would be still optimized for simplicity (and fitness to observational data of course), but besides that it would be optimized also for preference-to-necessity complexity ratio, so the additional principle would sound: what can be explained by preference, don't explain by necessity.
The goal of this change is to create some optimization proto-criterion for selecting “better” states. And I assume the better states are the more preferred states.
And the whole morality would be some (positive-) justice principle across all fundamental building blocks according to that proto-criterion.
But to keep this post short, I will end here. But since this is just an idea, please treat it accordingly.