rvnnt

LESSWRONG
LW

rvnnt — LessWrong

Replying to[Hiring] Principia Research Fellows

How would you rate Principia's operational adequacy? In particular: how would you rate Principia on closure and opsec? ^[1]

Alternatively, if you use a different framework for thinking about operational adequacy, I'd be very curious to read your assessment w.r.t. that framework! ↩︎

Replying toDesign sketches for a more sensible world

rvnnt5d

Design sketches for a more sensible world

I think something like what you're sketching here, viz "harnessing technology to make people and civilization saner", is probably highly valuable and possibly quite neglected. ^[1] Thank you for working on this.

A class of infrastructure/technologies that seem very important, but which I didn't see mentioned in this post: infrastructure for creating common knowledge of better equilibria and coordinating transitions to them. ^[2] Do you (have plans to) address anything matching that (vague) description anywhere?

Low-hanging dignity points! ↩︎
I.e., something that would solve problems of form "there exists a much better equilibrium, but getting there would

rvnnt8d

The Simplest Case for AI Catastrophe

There’s some chance that the patchwork AI safety strategy of the leading companies might just work well enough

I wonder if it'd be a good idea to mention that "a small number of humans have intent-aligned vastly superhuman AIs" does not yet imply "things will be nice (or tolerable) for most people"?

OTOH, then you maybe also need to point out that "superhuman AIs are open-weights and anyone can run them" also leads to death-or-worse (to pre-empt "solutions" like "so let's give everyone AI!")?

Replying toWhat do you do when you find out you have inconsistent probabilities?

rvnntJan 21, 2026

What do you do when you find out you have inconsistent probabilities?

The four probabilities given as premises are inconsistent. The first three determine the fourth. (Also, there's an arithmetic error in the p(G|M) calculation, as pointed out by Bucky.)

Given

p(M) = 0.9
p(G) = 0.05
p(M|G) = 0.99

it must be that

p(M|-G) = (p(M) - p(M,G)) / (1 - p(M,G) - p(-M,G)) = 0.8505 / 0.95

which is approximately 0.895263. Not 0.02.

If this feels confusing, I suggest drawing a Venn diagram or something. If you have a box of area 1.0, containing a blob M of area 0.9 and another blob G of area 0.05, such that G is almost entirely inside M, then...

rvnnt24d

Interesting. Thanks. How did you arrive at the above picture? Any sources of information you'd recommend in particular?

rvnnt25dQuick Take

After reading about Trump's actions w.r.t. Greenland, I'm updating further away from

Trump's policies/actions are at least partially aimed at pursuing the national interests of the US (albeit possibly in a very misguided/incompetent way)

and further in favor of both

Trump has gone insane,
and/or Trump is intentionally acting on behalf of Putin.

I'd like to find more/better sources of evidence about "what is the US executive branch optimizing for?"; curious to hear suggestions.

(Also, to Americans: How high/low salience is the issue in the US? Also: curious to read your analysis of your chief executive's behavior.)

rvnnt2mo

Dominic Cummings (former Chief Adviser to the UK PM) has written some things about nuclear strategy and how it's implemented in practice. IIUC, he's critical of (i.a.) how Schelling et al.'s game-theoretic models are (often naively/blindly) applied to the real world.

Replying toA friction in my dealings with friends who have not yet bought into the reality of AI risk

rvnnt2mo

A friction in my dealings with friends who have not yet bought into the reality of AI risk

I updated a bit towards thinking that incompetence-at-reasoning is a more common/influential factor than I previously thought. Thanks.

However: Where do you think that moral realism comes from? Why is it a "thorny" issue?

Replying toA Full Epistemic Stack: Knowledge Commons for the 21st Century

rvnnt2mo

A Full Epistemic Stack: Knowledge Commons for the 21st Century

social-media-like interfaces for uncovering group wisdom and will at larger scales while eliciting more productive discourse

That seems like it might significantly help with raising the sanity waterline, and thus help with coordinating on AI x-risk, and thus be extremely high-EV (if it's successfully implemented, widely adopted, and humanity survives for a decade or two beyond widespread adoption). ^[1]

Do you think it would be practically possible with current LLMs to implement a version of social media that promoted/suggested content based on criteria like

validity of reasoning,
concrete factual claims (whether correct or incorrect), vs vibing,
thoughtful long-form writing, vs memeing,
good-faith debate, vs e.g. adversarial tribalistic signaling?

The "widely adopted" part seems difficult to achieve, though. The hypermajority of genpop humans would probably just keep scrolling TikTok and consuming outrage porn on X, even if Civilization 2.0 Wholesome Social Media were available. ↩︎

Replying toA Full Epistemic Stack: Knowledge Commons for the 21st Century

rvnnt2mo

A Full Epistemic Stack: Knowledge Commons for the 21st Century

This discourse structure associates related claims and evidence, [...]

To make it practically possible for non-experts to efficiently make sense of large, spread-out collections of data (e.g. to answer some question about the discourse on some given topic), it's probably necessary to not only rapidly summarize all that data, but also translate it into some easily-human-comprehensible form.

I wonder if it's practically possible to have LMs read a bunch of data (from papers to Twitter "discourse") on a given topic, and rapidly/on-demand produce various kinds of concise, visual, possibly interactive summaries of that topic? E.g. something like this, or a probabilistic graphical model, or some kind of data visualization (depending on what aspect of what kind of topic is in question)?

rvnnt's Shortform

rvnnt

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

A potentially somewhat important thing which I haven't seen discussed:

People who have a lot of political power or own a lot of capital, are unlikely to be adversely affected if (say) 90% of human labor becomes obsolete and replaced by AI.
In fact, so long as property rights are enforced, and humans retain a monopoly on decisionmaking/political power, such people are not-unlikely to benefit from the economic boost that such automation would bring.
Decisions about AI policy are mostly determined by people with a lot of capital or political power. (E.g. Andreessen Horowitz, JD Vance, Trump, etc.)

(This looks like a decisionmaker is not the beneficiary -type of situation.)

Why does that matter?

It has implications for

rvnnt

This question is partly motivated by observing recent discussions about corrigibility and wondering to what extent the people involved have thought about how their results might be used.

If there existed practically implementable ways to make AGIs corrigible to arbitrary principals, that would enable a wide range of actors to eventually control powerful AGIs. Whether that would be net good or bad on expectation would depend on the values/morality of the principals of such AGIs.

Currently it seems highly unclear what kinds of people we should expect to end up in control of corrigible ASIs, if corrigibility were practically feasible.

What (crucial) considerations should one take into account, when deciding whether to publish---or with whom to privately share---various kinds of corrigibility-related results?

Requesting feedback/advice: what Type Theory to study for AI safety?

rvnnt

TL;DR:

Requesting feedback or advice on questions of what math to study (or what projects to engage in) in order to learn how to prove things about programs and to analyze the difficulty/possibility of proving things about programs. Ulterior motive: AI safety (and also progress towards a university degree).

Motivation, Goal, Background

To my current understanding, any endeavor to figure out how to program^[1] safe AI would probably eventually^[2] benefit a lot from an ability to prove various properties of programs:
The AI will have to self-modify, or create successor agents, i.e. (re)write (parts of) new AI-software; and we'd probably want the AI to be able to prove^[3] that said AI-software satisfies various "safety properties" (which

... (read 866 more words →)

LESSWRONG
LW

LESSWRONG
LW

rvnnt

rvnnt's Shortform

Should you publish solutions to corrigibility?

Requesting feedback/advice: what Type Theory to study for AI safety?

rvnnt

rvnnt

rvnnt's Shortform

Should you publish solutions to corrigibility?

Requesting feedback/advice: what Type Theory to study for AI safety?

TL;DR:

Motivation, Goal, Background