Riccardo Volpato

Once I was talking with a friend about an upcoming career decision. I said I wanted to do something "valuable" and he said that he recognises value when he sees it. Understanding values has never been easy for me, so I was intrigued by his sentence. I started asking myself some questions. Why doing something valuable felt much harder to me than to him? How did our approach to understanding what is valuable differed? Is determining value an objectively hard problem or not?

To start, I reviewed and reflected on some common ways in which we attempt to understand what we and others value.

Common ways of understanding value

Words

How many times when you asked... (read 1907 more words →)

Replying tointerpreting GPT: the logit lens

interpreting GPT: the logit lens

Apparently it is keeping around a representation of the token "plasma" with enough resolution to copy it . . . but it only retrieves this representation at the end! (In the rank view, the rank of plasma is quite low until the very end.)
This is surprising to me. The repetition is directly visible in the input: "when people say" is copied verbatim. If you just applied the rule "if input seems to be repeating, keep repeating it," you'd be good. Instead, the model scrambles away the pattern, then recovers it later through some other computational route.

One more reason on why this is suprising, is that other experiments found that this behaviour (forgetting... (read more)

Is 'satificing' optimisation?

It seems to me that the behavioural science research around 'the trade-off between maximising and satisficing' in terms of well-being relates with human beings being 'optimisers' only sometimes. Also, it seems to suggests that there can be alternative and better approaches to optimisation? Is this true? Or is 'satisficing' just a hidden multi-objective optimisation that factors in time and effort costs?

See what trigger my question: A decision-maker’s dilemma: search for the best option or settle for ‘good enough’?

Replying toInterpretability in ML: A Broad Overview

Interpretability in ML: A Broad Overview

I've spent more of my time thinking about the technical sub-areas, so I'm focused on situations where innovations there can be useful. I don't mean to say that this is the only place where I think progress is useful.

That seems more than reasonable to me, given the current state of AI development.

Thanks for sharing your reflections on my comment.

Replying toInterpretability in ML: A Broad Overview

Interpretability in ML: A Broad Overview

While I agree with you that setting the context as Safety narrows down the requirements space for interpretability, I think there could be more than just excluding the interpretability of non-technical users from the picture. The inspections that technicians would want to be reassured about the model safety are probably around its motivations (e.g is the system goal directed, is the system producing mesa-optimisers). However, it is still unclear to me how this relate with other interpretability desiderata you present in the post.

Plus, one could also immagine safety-relevant scenarios were it may be necessary or useful for non-technical users to be able to interpret the model. For instance, if the model has been deployed and is adaptive and we somehow cannot do this automatically, we would probably want users to be able to inspect if the system is somehow made a decision for the wrong reason.

Replying toInterpretability in ML: A Broad Overview

Interpretability in ML: A Broad Overview

Thanks for the in-depth post on topic. Your last paragraph on Utility is thought-provoking to say the least. I have seen a lot of work claiming to make models interpretable - and factually doing so as well - about which I felt an itch I could not fully verbalise. I think your point on Utility puts the finger on it: most of these works were technically interpreting the model but not actually useful to the user.

From this, we can also partially explain the current difficulties around "find better ways to formalize what we mean by interpretability". If acceptable interpretability depend on its usefulness, then it becomes context-dependent, which blows up the complexity of the attempt.

Hence Interpretability seems to be an interdisciplinary problem. One that requires having the rigth culture and principles to adopt the right interpretability tools for the right problem. This seems to be confirmed by the many contributions you shared from the social sciences on the topic.

What do you think of this perspective?

Replying toHuman-AI Interaction

Human-AI Interaction

I think we raise children to satisfy our common expected wellbeing (our + theirs + the overall societal one). Thus, the goal-directness comes from society as a whole. I think there is a key difference between this system and one where a a smarter-than-human AI focuses solely on the well-being of its users, even if it does Context Etrapolated Volition, which I think is what you are referring to when you talk about expected well being (which I agree that if you look only at their CEV-like property the two systems are equivalent).

Replying toHuman-AI Interaction

Human-AI Interaction

The problem with this line of reasoning is that it assumes that the goal-directness comes from the smarter part of the duo decision-maker and bearer of consequences. With children and animals we consider they preferences as an input into our decision making, which mainly seeks to satisfies our preferences. We do not raise children solely for the purpose of satisfying their preferences.

This is why Rohin stresses particuarly on the idea that the danger in is the source of goal-directedness and if it comes from humans, then we are safer.

Replying toAkrasia is confusion about what you want

Akrasia is confusion about what you want

Helpful post - thanks for writing it. From a phenomenological perspective, how can we reason well about the truth of this kind of "principles" (i.e. dual-model where S2 is better than S1 being less effective at dealing with motivational conflicts than than the perspective-shift you suggest) that are to some extent non-falisfiable?

Replying toAkrasia is confusion about what you want

Akrasia is confusion about what you want

This seems true to me (that it happens all the time). I think the article helps by showing that we often fail to recognise that A) and B) can both be true. Also, if we accept that A) and B) are both true and don't create an identify conflict about it, we can probably be more effective in striking a compromise (i.e. giving up either or finding some other way to get A that does not involve B).

Replying toIntuitions about goal-directed behavior

Intuitions about goal-directed behavior

My rough mental summary of these intuitions

Generalisation abilities suggests that behaviour is goal directed beucase it demonstrates adaptability (and goals are more adatable/compact ways of defining behaviour than others, like enumeration)
Powergrabs suggest that behaviour is goal directed beucase it reveals instrumentalism
Our understanding of intelligence migth be limited to human intelligence which is sometime goal directed so we use this a proxy of intelligence (adding some, perhaps unrefutable, skepticism of goal directness as a model of intelligence)

Research ideas to study humans with AI Safety in mind

Premise

Recently I spent some time thinking about ways in which studying the human side of human-machine systems would be beneficial to build aligned AIs. I discussed these ideas informally and people seemed interested and wanted to know more. Thus, I decided to write a list of research directions for studying humans that could help solve the alignment problem.

The list is non-exhaustive. Also, the intention behind it is not to argue that these research directions are more important than any other but rather to suggest directions to someone with a related background or personal fit in studying humans. There is also a lot of valuable work in AI Strategy that involves studying humans,

... (read 1454 more words →)

Replying toRisks from Learned Optimization: Conclusion and Related Work

Risks from Learned Optimization: Conclusion and Related Work

we currently don't have a formal specification of optimization

This seems to me a singificant bottleneck for progress. No formal specification of what optimisation is has been tried before? What has been achieved? Is anyone working on this?

What messy problems do you see Deep Reinforcement Learning applicable to?

What is the relationship between Preference Learning and Value Learning?

Many consider Deep Reinforcement Learning (DeepRL) systems, such as AlhpaZero or MuZero, as the most-likely early version of AGI systems.

However, a common critique to RL systems is that they are applicable only to well-defined problems, such as games.

Hence, a natural question that follow from this is: what "messy" problems could be tackled using DeepRL systems?

Here I intend messy as a pseudo-definition of problems that do not have clearly identifiable inputs and outputs. Examples that come to my mind are generally problems from Economics, Management and Policy Making.