LESSWRONG
LW

Vanessa Kosoy — LessWrong

I like the overall vibe. Two issues:

It says "Top Posts" and the mouse-over text is "by karma", however in reality I can choose which posts to put there. Now, I like it that I can choose which posts to put there, but once I customized them, the mouse-over becomes a lie.
~~The "recent comments" disappeared. This is~~ ~~really bad~~ ~~because I use that to find my recent comments when I want to edit them. (For example now I wanted to find this comment to add this second bullet but had to do it manually.)~~ OK, I now see I can find them under "feed" but this might be confusing.

Replying to"The first two weeks are the hardest": my first digital declutter

"The first two weeks are the hardest": my first digital declutter

[Context: I'm not a digital minimalist but I am somewhat of a "digital reducetarian": I don't have social media (besides LinkedIn) and have a browser plugin that reduces my access to particular websites (like LessWrong).]

Cool post :)

For me, there's something "strange" here (not surprising, but unlike my own experience), where the implication is that people have huge swaths of "free time" that they use for scrolling and the like (which you instead use for what's described in this post). I spend the vast majority of my time either working or doing something with kids/lovers/friends. (I did read this post in bed preparing to start my day, and am sneaking in this comment... (read more)

Apply to Vanessa's mentorship at PIBBSS

Vanessa Kosoy

1mo

The applications for the PIBBSS summer fellowship 2026 are now open, and I will be one of the mentors. If you want to work with me on the Learning-Theoretic AI Alignment Agenda (LTA), I recommend to apply. Exceptional fellows might be offered an (additional) paid research fellowship at CORAL after the programme, depending on available funding and other circumstances.

Why Apply?

I believe that the risk of a global catastrophe due to unaligned artificial superintelligence is the most pressing problem of our time. I also believe that the LTA is our last best chance for solving the technical alignment problem. Even if governance efforts succeed, they will only buy time: but we still need... (read 391 more words →)

Replying toMany arguments for AI x-risk are wrong

Vanessa Kosoy1mo

Many arguments for AI x-risk are wrong

A metacognitive agent is not really an RL algorithm is in the usual sense. To first approximation, you can think of it as metalearning a policy that approximates (infra-)Bayes-optimality on a simplicity prior. The actual thing is more sophisticated and less RL-ish, but this is enough to understand how it can avoid wireheading.

Many alignment failure modes (wireheading, self-modification, inner misalignment...) can be framed as "traps", since they hinge of the non-asymptotic properties of generalization, i.e. on the "prior" or "inductive bias". Therefore frameworks that explicitly impose a prior (such as a metacognitive agents) are useful for understanding and avoiding these failure modes. (But, this has little to do with the OP, the way I see it.)

Replying toMany arguments for AI x-risk are wrong

Vanessa Kosoy1mo

Many arguments for AI x-risk are wrong

The problem is, even if the argument that wireheading doesn't happen is valid, it is not a cancellation. It just means that the wireheading failure mode is replaced by some equally bad or worse failure mode. This argument is basically saying "your model that predicted this extremely specific bad outcome is inaccurate, therefore this bad outcome probably won't happen, good news!" But it's not meaningful good news, because it does literally nothing to select for the extremely specific good outcome that we want.

If a rocket was launched towards the Moon with a faulty navigation system, that does not mean the rocket is likely to land on Mars instead. Observing that the navigation... (read more)

Replying toMany arguments for AI x-risk are wrong

Vanessa Kosoy1mo

Many arguments for AI x-risk are wrong

What do you mean "randomly come upon A"? RL is not random. Why wouldn't it find A?

Let the proxy reward function we use to train the AI be and the "true" reward function that we intend the AI to follow be $r_{t}$ . Supposedly, these function agree on some domain $D$ but catastrophically go apart outside of it. Then, if all the training data lies inside $D$ , which reward function is selected depends on the algorithm's inductive bias (and possibly also on luck). The "cancellation" hope is then that inductive bias favors $r_{t}$ over $r_{p}$ .

But why would that be the case? Realistically, the inductive bias is something like "simplicity". And human preferences are very complex. On the other hand, something like "the reward is such-and-such bits in the input" is very simple. So instead of cancelling out, the problem is only aggravated.

And that's under the assumption that $r_{p}$ and $r_{t}$ actually agree on $D$ , which is in itself wildly optimistic.

Replying toDefining alignment research

Vanessa Kosoy1moReview for 2024 Review

Defining alignment research

In the post Richard Ngo talks about delineating "alignment research" vs. "capability research", i.e. understanding what properties of technical AI research make it, in expectation, beneficial for reducing AI-risk rather than harmful. He comes up with a taxonomy based on two axes:

Cognitivist vs. Behaviorist, i.e. focused on internals vs. external behavior. Arguably, net-beneficial research tends to be on the cognitivist side.
Worst-case vs. Average-case, i.e. focused on rare failures vs. "usual" behavior. Arguably, net-beneficial research tends to be on the worst-case side.

I think that Ngo raises an important question, and his answers are pointing in the right way. On my part, I would like to slightly reframe his parameters and add two more... (read 645 more words →)

Replying toOverwhelming Superintelligence

Vanessa Kosoy1mo

Overwhelming Superintelligence

I am separately worried about "Carefully Controlled Moderate Superintelligences that we're running at scale, each instance of which is not threatening, but, we're running a lot of them...

I think that this particular distinction is not the critical one. What constitutes an "instance" is somewhat fuzzy. (A single reasoning thread? A system with a particular human/corporate owner? A particular source code? A particular utility function?) I think it's more useful to think in terms of machine intelligence suprasystems with strong internal coordination capabilities. That is, if we're somehow confident that the "instances" can't or won't coordinate either causally or acausally, then they are arguably truly "instances", but the more they can coordinate the more we should be thinking of them in the aggregate. (Hence, the most cautious risk estimate comes from comparing the sum total of all machine intelligence against the sum total of all human intelligence^[1].)

^{^}
More precisely, not even the sum total of all human intelligence, but the fraction of human intelligence that humans can effectively coordinate. See also comment by Nisan.

Replying toOverwhelming Superintelligence

Vanessa Kosoy1mo

Overwhelming Superintelligence

There seem to be two underlying motivations here, which are best kept separate.

One motivation is having a good vocabulary to talk about fine-grained distinctions. I'm on board with this one. We might want to distinguish e.g.:

Smarter than a median human along all AI-risk-relevant axes
Smarter than the smartest human along all AI-risk-relevant axes
Smarter than all of humanity put together along all AI-risk-relevant axes
Smart enough to have a 50% success probability to kill all humans if it chooses to, given current level of countermeasures
Smart enough to have a 50% success probability to kill all humans if it chooses to, even if best-case countermeasures are in place (this particular distinction inspired by Buck's comments on

Vanessa Kosoy1moReview for 2024 Review

My AGI safety research—2024 review, ’25 plans

This post is an overview of Steven Byrnes' AI alignment research programme, which I think is interesting and potentially very useful.

In a nutshell, Byrnes' goal is to reverse engineer the human utility function, or at least some of its central features. I don't think this will succeed in the sense of, we'll find an explicit representation that can be hard-coded into AI. However, I believe that this kind of research is useful for two main reasons:

Bridging brain science and agent theory is a promising way to make sure that we build a theory of agents broad enough to include humans. The latter is crucial in order to formally define alignment (since alignment

Vanessa Kosoy1mo

Overwhelming Superintelligence

Why are we giving up on plain "superintelligence" so quickly? According to Wikipedia:

A superintelligence is a hypothetical agent that possesses intelligence surpassing that of the most gifted human minds. Philosopher Nick Bostrom defines superintelligence as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest".

According to Google AI Overview:

Superintelligence (or Artificial Superintelligence - ASI) is a hypothetical AI that vastly surpasses human intellect in virtually all cognitive domains, possessing superior scientific creativity, general wisdom, and social skills, operating at speeds and capacities far beyond human capability, and potentially leading to profound societal transformation or existential risks if not safely aligned with human goals.

I don't think I saw anyone use "superintelligence" as "better than a majority of humans on some specific tasks" before very recently. (Was DeepBlue a superintelligence? Is a calculator superintelligence?)

Lectures on statistical learning theory for alignment researchers

Vanessa Kosoy

5mo

This is a recorded series of lectures and homework assignments discussions on the topic of statistical learning theory, co-organized by Gergely Szucs (@Yegreg) and @Alex Flint. The homework assignments are available here, and lecture notes are available here.

The lectures are intended for those who want to do research on the learning-theoretic agenda and need to catch-up on the necessary background knowledge. As complementary resources, consider looking into my MATS lectures, the LTA reading list and the (not quite finished) sequence on infra-Bayesianism by @Brittany Gelb.

If you feel ready to start with actual research, consider applying to my track in the PIBBSS fellowship.

Privacy note: According to what I know, everyone who appear in the recording consented to the recording becoming available online. However, if there was some mistake, s.t. you are in the recording but don't want to be there, DM me and I will take it down ASAP.

H/t to my beloved spouse @Marcus Ogren for placing the lectures on YouTube.

It occurred to me that one major success story of banning a technology (whether justifiably or not), is the laws against genetic engineering and cloning in humans. It makes me wonder what can we learn from that, which is applicable to promoting a global moratorium on ASI.

New Paper: Ambiguous Online Learning

Vanessa Kosoy

8mo

Abstract: We propose a new variant of online learning that we call "ambiguous online learning". In this setting, the learner is allowed to produce multiple predicted labels. Such an "ambiguous prediction" is considered correct when at least one of the labels is correct, and none of the labels are "predictably wrong". The definition of "predictably wrong" comes from a hypothesis class in which hypotheses are also multi-valued. Thus, a prediction is "predictably wrong" if it's not allowed by the (unknown) true hypothesis. In particular, this setting is natural in the context of multivalued dynamical systems, recommendation algorithms and lossless compression. It is also strongly related to so-called "apple tasting". We show that... (read 250 more words →)

Master post for multi-agency. See also.

New Paper: Infra-Bayesian Decision-Estimation Theory

Vanessa Kosoy

Vanessa Kosoy, Diffractor

10mo

Diffractor is the first author of this paper.
Official title: "Regret Bounds for Robust Online Decision Making"

Abstract: We propose a framework which generalizes "decision making with structured observations" by allowing robust (i.e. multivalued) models. In this framework, each model associates each decision with a convex set of probability distributions over outcomes. Nature can choose distributions out of this set in an arbitrary (adversarial) manner, that can be nonoblivious and depend on past history. The resulting framework offers much greater generality than classical bandits and reinforcement learning, since the realizability assumption becomes much weaker and more realistic. We then derive a theory of regret bounds for this framework. Although our lower and upper bounds

... (read 178 more words →)

[Closed] Gauging Interest for a Learning-Theoretic Agenda Mentorship Programme

Vanessa Kosoy

I'm planning to organize a mentorship programme for people who want to become researchers working on the Learning-Theoretic Agenda (LTA). I'm still figuring out the detailed plan, the logistics and the funding, but here's an outline of how it would looks like. ~~To express interest, submit~~ ~~this form~~. [EDIT: The programme described here will not happen this year, due to lack of funding. If you're interested in my mentorship, consider applying to the PIBBSS Fellowship.]

Why Apply?

Master post for selection/coherence theorems. Previous relevant shortforms: learnability constraints decision rules, AIT selection for learning.

Video lectures on the learning-theoretic agenda

Vanessa Kosoy

This is a YouTube playlist of recorded lectures on the learning-theoretic AI alignment agenda (LTA) I gave for my MATS scholars of the Winter 2024 cohort, edited by my beloved spouse @Marcus Ogren. H/t William Brewer for helping with the recording, and the rest of the MATS team for making this possible.

I hope these will become a useful resource for anyone who wants to get up to speed on the LTA, complementary to the reading list. Notable topics that aren't covered include metacognitive agents (although there is an older recorded talk on that) and infra-Bayesian physicalism. In the future, I might record more lectures to expand this playlist.

EDIT: I know the audio... (read more)

I just read Daniel Boettger's "Triple Tragedy And Thankful Theory". There he argues that the thrival vs. survival dichotomy (or at least its implications on communication) can be understood as time-efficiency vs. space-efficiency in algorithms. However, it seems to me that a better parallel is bandwidth-efficiency vs. latency-efficiency in communication protocols. Thrival-oriented systems want to be as efficient as possible in the long-term, so they optimize for bandwidth: enabling the transmission of as much information as possible over any given long period of time. On the other hand, survival-oriented systems want to be responsive to urgent interrupts which leads to optimizing for latency: reducing the time it takes between a piece of information appearing on one end of the channel and that piece of information becoming known on the other end.

Here's the sketch of an AIT toy model theorem that in complex environments without traps, applying selection pressure reliably produces learning agents. I view it as an example of Wentworth's "selection theorem" concept.

Consider any environment $μ$ of infinite Kolmogorov complexity (i.e. uncomputable). Fix a computable reward function

r : (A \times O)^{*} \to [0, 1]

Suppose that there exists a policy $π^{*}$ of finite Kolmogorov complexity (i.e. computable) that's optimal for $μ$ in the slow discount limit. That is,

lim γ \to 1 (1 - γ) (max π E_{μ π} [\infty \sum n = 0 γ^{n} r_{n}] - E_{μ π^{*}} [\infty \sum n = 0 γ^{n} r_{n}]) = 0

Then, $μ$ cannot be the only environment with this property. Otherwise, this property could be used to define $μ$ using a finite number of bits, which is impossible^[1]. Since $μ$ requires infinitely many more bits to specify than $π^{*}$ and $r$ , there has to be infinitely many environments with the same property^[2]. Therefore, $π^{*}$ is a reinforcement learning algorithm for... (read more)

The recent success of AlphaProof updates me in the direction of "working on AI proof assistants is a good way to reduce AI risk". If these assistants become good enough, it will supercharge agent foundations research^[1] and might make the difference between success and failure. It's especially appealing that it leverages AI capability advancement for the purpose of AI alignment in a relatively^[2] safe way, thereby the deeper we go into the danger zone the greater the positive impact^[3].

EDIT: To be clear, I'm not saying that working on proof assistants in e.g. DeepMind is net positive. I'm saying that a hypothetical safety-conscious project aiming to create proof assistants for agent foundations research, that neither leaks dangerous knowledge nor repurposes it for other goals, would be net positive.

^{^}
Of course, agent foundation research doesn't reduce to solving formally stated mathematical problems. A lot of it is searching for the right formalizations. However, obtaining proofs is a critical arc in the loop.
^{^}
There are some ways for proof assistants to feed back into capability research, but these effects seem weaker: at present capability advancement is not primarily driven by discovering theorems, and if this situation changes it would mean we now actually know something about what we're doing, which would be great news in itself.
^{^}
Until we become saturated on proof search and the bottlenecks are entirely elsewhere.

Linear infra-Bayesian Bandits

Vanessa Kosoy

Linked is my MSc thesis, where I do regret analysis for an infra-Bayesian^[1] generalization of stochastic linear bandits.

The main significance that I see in this work is:

Expanding our understanding of infra-Bayesian regret bounds, and solidifying our confidence that infra-Bayesianism is a viable approach. Previously, the most interesting IB regret analysis we had was Tian et al which deals (essentially) with episodic infra-MDPs. My work here doesn't supersede Tian et al because it only talks about bandits (i.e. stateless infra-Bayesian laws), but it complements it because it deals with a parameteric hypothesis space (i.e. fits into the general theme in learning-theory that generalization bounds should scale with the dimension of the hypothesis class).
Discovering some

Vanessa Kosoy

Vanessa Kosoy, rosiecam

The beauty industry offers a large variety of skincare products (marketed mostly at women), differing both in alleged function and (substantially) in price. However, it's pretty hard to test for yourself how much any of these product help. The feedback loop for things like "getting less wrinkles" is very long.

So, which of these products are actually useful and which are mostly a waste of money? Are more expensive products actually better or just have better branding? How can I find out?

I would guess that sunscreen is definitely helpful, and using some moisturizers for face and body is probably helpful. But, what about night cream? Eye cream? So-called "anti-aging"? Exfoliants?

122

People like Andrew Critch and Paul Christiano have criticized MIRI in the past for their "pivotal act" strategy. The latter can be described as "build superintelligence and use it to take unilateral world-scale actions in a manner inconsistent with existing law and order" (e.g. the notorious "melt all GPUs" example). The critics say (justifiably IMO), this strategy looks pretty hostile to many actors and can trigger preemptive actions against the project attempting it and generally foster mistrust.

Is there a good alternative? The critics tend to assume slow-takeoff multipole scenarios, which makes the comparison with their preferred solutions to be somewhat "apples and oranges". Suppose that we do live in a hard-takeoff singleton... (read 421 more words →)

Ratfic idea / conspiracy theory: Yudkowsky traveled back in time to yell at John Nash about how Nash equilibria are stupid^[1], and that's why Nash went insane.

h/t Marcus (my spouse)

^{^}
They are.

AI Alignment Metastrategy

Vanessa Kosoy

I call "alignment strategy" the high-level approach to solving the technical problem^[1]. For example, value learning is one strategy, while delegating alignment research to AI is another. I call "alignment metastrategy" the high-level approach to converging on solving the technical problem in a manner which is timely and effective. (Examples will follow.)

In a previous article, I summarized my criticism of prosaic alignment. However, my analysis of the associated metastrategy was too sloppy. I will attempt to somewhat remedy that here, and also briefly discuss other metastrategies, to serve as points of contrast and comparison.

Conservative Metastrategy

The conservative metastrategy follows the following algorithm:

As much as possible, stop all work on AI capability outside of

... (read 1845 more words →)

126

•••

Critical review of Christiano's disagreements with Yudkowsky

Vanessa Kosoy

This is a review of Paul Christiano's article "where I agree and disagree with Eliezer". Written for the LessWrong 2022 Review.

In the existential AI safety community, there is an ongoing debate between positions situated differently on some axis which doesn't have a common agreed-upon name, but where Christiano and Yudkowsky can be regarded as representatives of the two directions^[1]. For the sake of this review, I will dub the camps gravitating to the different ends of this axis "Prosers" (after prosaic alignment) and "Poets"^[2]. Christiano is a Proser, and so are most people in AI safety groups in the industry. Yudkowsky is a ~~typical Poet~~ [sort-of Poet, but there's an important departure... (read 4211 more words →)

176

•••

Here is a way to construct many learnable undogmatic ontologies, including such with finite state spaces.

A deterministic partial environment (DPE) over action set $A$ and observation set $O$ is a pair $(D, ϕ)$ where $D \subseteq (O \times A)^{*}$ and $ϕ : D \to O$ s.t.

If $h \in (O \times A)^{*}$ is a prefix of some $g \in D$ , then $h \in D$ .
If $h, g \in D$ , $p \in O$ and $h p$ is a prefix of $g$ , then $ϕ (h) = p$ .

DPEs are equipped with a natural partial order. Namely, $(D, ϕ) \leq (E, ψ)$ when $D \subseteq E$ and $ϕ = ψ |_{D}$ .

Let $S$ be a strong upwards antichain in the DPE poset which doesn't contain the bottom DPE (i.e. the DPE with $D = \emptyset$ ). Then, it naturally induces an infra-POMDP. Specifically:

The state space is $S$ .
The initial infradistribution is $⊤_{S}$ .
The observation mapping is $ω (D, ϕ) := ϕ (ϵ)$ , where $ϵ$ is the empty history.
The transition infrakernel is $T (D, ϕ; a) := ⊤_{N (D, ϕ; a)}$ , where

N (D, ϕ; a) := {(E, ψ) \in S | \forall h \in (O \times A)^{*} : ϕ (ϵ) a h \in D ⟹ h \in E \land ψ (h) = ϕ (ϕ (ϵ) a h)}

If $N (D, ϕ; a)$ is non-empty for all $(D, ϕ) \in S$ and $a \in A$ , this is a learnable undogmatic ontology.

Any $n \in N$ yields an example $S_{n}$ . Namely, $(D, ϕ) \in S_{n}$ iff $D \neq \emptyset$ and for any $h \in D$ it holds that:

$| h | \leq n$
If $| h | < n$ then for any $a \in A$ , $h a ϕ (a) \in D$ .

I think that for ~~any continuous~~ some non-trivial... (read more)

Master post for ideas about infra-Bayesianism.

LESSWRONG
LW

LESSWRONG
LW

Vanessa Kosoy

The Learning-Theoretic Agenda: Status 2023

Infra-Bayesian physicalism: a formal theory of naturalized induction

Critical review of Christiano's disagreements with Yudkowsky

Video lectures on the learning-theoretic agenda

Vanessa Kosoy

Apply to Vanessa's mentorship at PIBBSS

Lectures on statistical learning theory for alignment researchers

New Paper: Ambiguous Online Learning

New Paper: Infra-Bayesian Decision-Estimation Theory

[Closed] Gauging Interest for a Learning-Theoretic Agenda Mentorship Programme

Video lectures on the learning-theoretic agenda

Linear infra-Bayesian Bandits

Vanessa Kosoy

The Learning-Theoretic Agenda: Status 2023

Infra-Bayesian physicalism: a formal theory of naturalized induction

Critical review of Christiano's disagreements with Yudkowsky

Video lectures on the learning-theoretic agenda

Vanessa Kosoy

Apply to Vanessa's mentorship at PIBBSS

Lectures on statistical learning theory for alignment researchers

New Paper: Ambiguous Online Learning

New Paper: Infra-Bayesian Decision-Estimation Theory

[Closed] Gauging Interest for a Learning-Theoretic Agenda Mentorship Programme

Video lectures on the learning-theoretic agenda

Linear infra-Bayesian Bandits

Why Apply?

Why Apply?

Conservative Metastrategy