LESSWRONG
LW

clem_acs — LessWrong

Between June and September 2024, we ran the third iteration of the PIBBSS Summer Research Fellowship. Here are our reflections on how the program went and what we learned. Apply for the 2025 program here!

TLDR:

The 2024 fellowship demonstrated continued success in attracting and developing senior academic talent while refining our approach to interdisciplinary AI safety research. Key outcomes include:

Successfully facilitating the transition of multiple early-to-mid-career researchers into AI safety
Strong research outputs, including advances in mechanistic interpretability that one mentor described as "a 70th percentile paper in mech interp"
Increased seniority of fellows led to more sophisticated and complete research outputs
Going forward, we plan to strengthen research support infrastructure and better balance research direction diversity,

... (read 965 more words →)

Replying toWhy I funded PIBBSS

clem_acs1y

Why I funded PIBBSS

Clem here - I was fellowship lead this year and have been a research affiliate and mentor for PIBBSS in the past. Thanks for posting this. As might be expected in my position, I'm much more bullish than you / most people on what is often called "blue sky" research. Breakthroughs in the our fundamental understanding of agency, minds, learning etc. seem valuable in a range of scenarios, not just in world dominated by an "intelligence explosion". In particular, I think that this kind of work (a) benefits a huge amount from close engagement with empirical work, and (b) itself is very likely to inform near-future prosaic work. Furthermore, I feel that... (read more)

Announcing the PIBBSS Symposium '24!

DusanDNesic

DusanDNesic, clem_acs

Tl;dr

We are pleased to invite you to the second PIBBSS Symposium, where the fellows from the ‘24 fellowship program present their work.

The symposium is taking place online, over several days in the week of September 9th.

Check out the full program, including brief descriptions for each talk. (You can toggle between the different days.)
Register via this link. You will receive an email with the event link which you can use for the entire symposium. You will be able to drop into as few or as many presentations and breakout sessions as you like.
Click here to add the schedule to your google calendar.

About PIBBSS

PIBBSS is a research initiative aiming to explore parallels between intelligent behavior... (read 647 more words →)

Replying to'Theories of Values' and 'Theories of Agents': confusions, musings and desiderata

clem_acs2y

'Theories of Values' and 'Theories of Agents': confusions, musings and desiderata

There is another sense in which I would not want to say that there is any particular hierarchy between natural/unnatural/rational constraints.

I think there's a lot to unpack here. I'm going to give it a preliminary go, anticipating that it's likely to be to a bit all over the place. The main thread I want to pull is what it means to impose a particular hierarchy between the constraints, and then see how this leads to many possible hierarchies in such a way that it feels like no particular hierarchy is privileged.

From a "natural" point of view, which privileges physical time, individuation is something that must be explained - a puzzle which is... (read more)

Replying to"Concepts of Agency in Biology" (Okasha, 2023) - Brief Paper Summary

clem_acs3y

"Concepts of Agency in Biology" (Okasha, 2023) - Brief Paper Summary

Motivation 1 ('organisms-as-agents thesis') "says that organisms really do exhibit some or even all of the attributes of agency". Motivation 2 ('organism-as-agents heuristic') "says that it can be heuristically useful to treat organisms as if they were agents for certain intellectual purposes".

It's interesting that both motivations appear to be about modelling organisms as agents, as opposed to any other level of organisation. This feels like it misses some of the most interesting insights we might get from biological agency, namely those around agents at different levels of organisation interacting - e.g: ants and ant colonies, cancer cells and multicellular organisms, or individual organisms and selection pressures (which could be treated as as-if agents at evolutionary timescales).

Contingency: A Conceptual Tool from Evolutionary Biology for Alignment

clem_acs

-1. Motivation

As AI systems become increasingly powerful, the chance of them developing dangerous features becomes increasingly likely. A key concern of AI alignment researchers is that there are dangerous features, such as power-seeking or situational awareness, which are convergent.^[1] This would mean that these capabilities would likely arise regardless of the particular trajectory AI development took. In a recent tweet, Eliezer Yudkowsky argues that this is the key question for AI alignment research:

> From my perspective, we are trying to evaluate a picture of convergent ruin rather than any pictures of contingent ruin. That is, I'm relatively less interested in contingent-doom scenarios like "What if superintelligence is easily alignable, but Greenpeace is first

... (read 4151 more words →)

Replying toLessons from Convergent Evolution for AI Alignment

clem_acs3y

Lessons from Convergent Evolution for AI Alignment

"Social" is slightly too coarse-grained a tag. The thing we're actually interested in is "whether successfully predicting the behaviour of other members of its own species is a strong selection pressure". Social collaboration is one way this happens - another seems to be "deception" arms races (such as corvids stealing and hiding things from each other), or specific kinds of mating rituals. It also depends on the relative strength of other selection pressures - in most cases highly intelligent creatures also seem to have developed a "slack" in resources they can devote to intelligence (e.g. humans cooking food).

This does seem to hold for cephalopods - a strong datapoint for which being their highly sophisticated forms of communication (e.g. video below).

The space of systems and the space of maps

Jan_Kulveit

Jan_Kulveit, rosehadshar, Nora_Ammann, clem_acs

When we're trying to do AI alignment, we're often studying systems which don't yet exist. This is a pretty weird epistemic activity, and seems really hard to get right. This post offers one frame for thinking about what we're actually doing when we're thinking about AI alignment: using parts of the space of maps to reason about parts of the space of intelligent systems.

In this post, we:

Introduce a simple model of the epistemic situation, and
Share some desiderata for maps useful for alignment.

We hope that the content is mostly the second kind of obvious: obvious once you see things in this way, which you maybe already do. In our experience, this comes with... (read 1248 more words →)

LESSWRONG
LW

LESSWRONG
LW

clem_acs

Retrospective: PIBBSS Fellowship 2024

Contingency: A Conceptual Tool from Evolutionary Biology for Alignment

The space of systems and the space of maps

Announcing the PIBBSS Symposium '24!

clem_acs

clem_acs

Retrospective: PIBBSS Fellowship 2024

Announcing the PIBBSS Symposium '24!

Contingency: A Conceptual Tool from Evolutionary Biology for Alignment

The space of systems and the space of maps

clem_acs

Retrospective: PIBBSS Fellowship 2024

Contingency: A Conceptual Tool from Evolutionary Biology for Alignment

The space of systems and the space of maps

Announcing the PIBBSS Symposium '24!

clem_acs

clem_acs

Retrospective: PIBBSS Fellowship 2024

Announcing the PIBBSS Symposium '24!

Contingency: A Conceptual Tool from Evolutionary Biology for Alignment

The space of systems and the space of maps

TLDR:

Tl;dr

About PIBBSS

-1. Motivation