There is ambiguity in "source of values" which this outline elides. There is source as "method of discovery" and source as "kind of fact that determines the truth value of normative statements" and source as "causal origin of value-based thinking and language". And there is no reason why the "source of values" should be the same for each of those readings. Though obviously they could be.

So under "conscious cognition": traditional Kantian deontology is discovered by conscious cognition but the facts that make it true are supposed to be external to human minds (purportedly they exist in reason itself, like mathematical truths). But there are definitely also people who hold the view that what makes a decision a moral one is just that it was arrived at by the properly and rationally reflecting on values.

I don't think any position can be accurately categorized without understanding how that position answers each of the above readings of "source". So for instance, my own position would be that the causal origin of values is a synthesis of culturally constructed intuitions and innate, evolved intuitions on which we perform conscious cognitive operations (which are themselves a synthesis of innate abilities and technologies). So a mix between conscious and unconscious cognition, other humans, and some degree of reason. I don't think there are any facts that make value language true and method of discovery is just what we think we're doing with the above. It's then helpful to ask something like: well how do you want a superintelligence to decide what to do? The answer to which somehow involves recursing the causal source of value judgments upon itself. E.g. It's the same as asking "How ought a super intelligence behave?" which is just an instance of a value judgment. But obviously it is really difficult to a) precisely define that process and b)determine how exactly that recursion ought to happen.

I thought "indirect normativity" was a general term due to Nick Bostrom meant to cover e.g. CEV among other proposals. Could be wrong.

It may be interesting to note that this outline implies that when we discuss questions like "What's your utility function?" or "Do humans have utility functions?" we should be careful to distinguish what kind of utility function we are talking about. Examples:

• a utility function that represents my revealed preferences
• the utility function implied by my consequentialist moral principles
• the utility function that corresponds to my intuitions about the desirabilities of various specific outcomes
• the utility function I actually use when I engage in explicit consequentialist reasoning
• the utility function I actually use when I engage in subconscious model-based decision making
• the utility function I would eventually decide upon if I thought about it for a long time
• the utility function that best captures my intuitions about what "my real values" means
• the utility function that represents my real values (this may seem equivalent to the one above, except that I don't seem to have clear intuitions in the matter, what intuitions I do have seem subject to change, and maybe there is a fact of the matter about what my real values are beyond my intuitions about it?)

I don't know what my values are. I don't even know how to find out what my values are.

I find this confusing. Could you please give a precise definition of "values" in this context?

When you say "values", do you mean instrumental values, or do you mean terminal values? If the former then the answer is simple. This is what we spend most of our time doing. Will tweaking my diet in this way cause me to have more energy? Will asking my friend in this particular way cause them to accept my request? Etc. This is as mundane as it gets.

If the latter, the answer is a bit more complicated, but really it shouldn't be all that confusing. As agents, we're built with motivation systems, where out of all possible sensory patterns, some present to us as neutral, others as inherently desirable, and the last subset as inherently undesirable. Some things can be more desirable or less desirable, etc., thus these sensory components each run on at least one dimension.

Sensory patterns that present originally as inherently neutral may either be left as irrelevant (these are the things put on auto-ignore, which are apt to make a return to one's conscious awareness if certain substances are taken, or if careful introspection is engaged in), or otherwise acquire a 'secondary' desirability or undesirability via being seen to be in causal connection with something that presents as inherently one way or the other, for example finding running enjoyable because of certain positive benefits acquired in the past from the activity.

Thus to discover one's terminal values, one must simply identify these inherently desirable sensory patterns, and figure out which ones would top the list as 'most desirable' (in terms of nothing other than how it strikes one's perception). A good heuristic for this would be to see what other people consider enjoyable or fun, and then try it, and see what happens, but at the same time making sure to disambiguate any identity issues from the whole thing, such as sexual hangups making one unable to enjoy something widely considered to have one of the strongest effects in terms of 'wanting to engage in this behavior because it's so great'--sexual or romantic interaction.

But at the most fundamental, there's nothing to the task of figuring out one's terminal values other than simply figuring out what sensory patterns are most 'enjoyable' in the most basic sort of way imaginable, on a timescale sufficiently long-term to be something one would be unlikely to refer to as 'akrasia'. Even someone literally physically unable to experience certain positive sensory patterns, such as someone with extremely low libido because of physiological problems, would most likely qualify as making a 'good choice' if they engage in a course of action apt to cause them to begin to be able to experience these sensory patterns, such as that person implementing a particular lifestyle protocol likely to fix their physiological issues and bring them libido to a healthy level.

It gets somewhat confusing when you factor in the fact that the sensory patterns one is able to experience can shift over time, such as libido increasing or decreasing, or going through puberty, or something like that, along with factoring in akrasia, and other problems that make us seem less 'coherent' of agents, but I believe all the fog can be cut through if one simply makes the observation that sensory patterns present to us as either neutral, inherently desirable, or inherently undesirable, and that the latter two run on a dimension of 'more or less'. Neutral sensory patterns acquire 'secondary' quality on these dimensions depending on what the agent believes to be their causal connections to other sensory patterns, each ultimately needing to run up against an 'inherently motivating' sensory pattern to acquire significance.

99% of values come from copied entities (genes, memes, etc). Basically, any non-trivial optimisation process (i.e. more advanced than exhaustive/random search) involves the copying-with-variation of previously-successful solutions. So: a useful way of classifing many values, is to consider what was copied in order to produce them.

Framing in terms of "[your] values" creates different emphases from framing in terms of "morality", and that makes it hard for me to comment. "Decision-policy-relevant stuff" would evoke still different intuitions. There might be a way to go meta here.

This outline seems like it could be very useful! I like this post.

I am confused about what is meant by 1.1 though. Are you referring to some hypothetical potentially real external agent who can tell us something about what to value? Or are you saying we can learn what someone values by observing their religious beliefs? Or simply that a person's religious beliefs inform their morals?