LESSWRONG
LW

James Payor — LessWrong

What interesting things do y'all think are up with AI lab politics these days? Also why is everyone (or just many people in these circle) going to Anthropic now?

Any changes in how things seem for control plans based on vibes and awareness present in more recent models? (GPT-5 series may not count here; I'm mostly interested in visiblity on the next generation that are coming, of which I think Opus 4.5 is a preview but I'm fairly unsure.)

Anything generally striking about how things look in the landscape and models versus a year ago?

James Payor15d

I would contest the frame here. In particular I think it won't hold up because things won't stay as capital-bound as they are now, and that seriously messes with the continuity required for today's investments to maintain their relative portion of the pie. What do you think of this part?

(EDIT: Okay I think you are referencing this sort of thing with "most people can't invest in assets that grow at the average rate", but I still take issue with some picture in which everything is apportioned to assets that grow or something like that.)

To expand: I think today's capital earns its gains mostly because it is a required input, which thereby gives... (read more)

Replying toWhat’s going on at CFAR? (Updates and Fundraiser)

James Payor1mo

What’s going on at CFAR? (Updates and Fundraiser)

I would like to thank you Anna for this write-up! I love the thoughtfulness and ideas on institutional design, relationship to donors and EA, and others. Much of this post was quite viscerally relaxing for me to read, in a "finally it feels possible to jointly understand X and Y and Z" kind of way, which I'm quite glad for.

•••

Replying toThe Weirdness of Dating/Mating: Deep Nonconsent Preference

James Payor1mo

The Weirdness of Dating/Mating: Deep Nonconsent Preference

On the overall note of the post:

I claim that most women have a “deep” preference for nonconsent in dating/mating. It’s not just a kink; from the first approach to a date to sex, women typically want to not have to consent to what’s happening.

...I would like to say that, in my capacity as just-some-guy, this has not been my lived experience interacting with women, and I disagree with a lot of the framing in this post. (I don't mean to contest what you've experienced; I would interpret these events pretty differently though.)

I'm writing this mostly because I'm finding it horrifying that it appears the LessWrong consensus reality is treating "nonconsent" stuff as a cornerstone of how it views women. This somehow seems to increasingly to be the case, seems wrong to me, and contributes to the erasure of some things I consider important.

Replying toWorking through a small tiling result

James Payor9mo

Working through a small tiling result

Meta note: Thanks for your comment! I failed to reply to this for a number of days, since I was confused about how to do that in the context of this post. Still though I think it's relevant about probabilistic reasoning, and I've now offered my thoughts in the other replies.

Replying toWorking through a small tiling result

James Payor9mo

Working through a small tiling result

Anyhow, regarding probability distributions, there's some philosophical difficulty in my opinion about "grounding". Specifically, what reason should I have to trust that the probability distribution is doing something sensible around my safety questions of interest? How did we construct things such that it was?

The best approach I'm aware of to building a computable (but not practical) distribution with some "grounding" results is logical induction / Garrabrant induction. They come with have a self-trust result of the form that logical inductors will, across time, converge to predicting their future selves' probabilities agree with their current probabilities. If I understand correctly, this includes limiting toward predicting a conditional probability $p$ for an event $X$ if we are given... (read more)

Replying toWorking through a small tiling result

James Payor9mo

Working through a small tiling result

In a more realistic and complicated setting, we may definitely want to be obtaining a high probability under some distribution we trust to be well-grounded, as our condition for a chain of trust. In terms of the technical difficulty I'm interested in working through, I think it should be possible to get satisfying results about proving that another proof system is correct, and whatnot, without needing to invoke probability distributions. To the extent that you can make things work with probabilistic reasoning, I think they can also be made to work in a logic setting, but we're currently missing some pieces.

Working through a small tiling result

James Payor

9mo

tl;dr it seems that you can get basic tiling to work by proving that there will be safety proofs in the future, rather than trying to prove safety directly.

"Tiling" here roughly refers to a state of affairs in which we have a program that is able to prove itself safe to run. I'll use a simple problem to keep this post self-contained, but here are some links to some relevant discussion from the past.

The idea I work through below is not new, here is Giles saying it 13 years ago. It's going to be brittle as well, but it seems to me like it's relevant to a general answer for tiling. I'd... (read 1285 more words →)

Replying to[Linkpost] "Governance of superintelligence" by OpenAI

James Payor3y

[Linkpost] "Governance of superintelligence" by OpenAI

I really should have something short to say, that turns the whole argument on its head, given how clear-cut it seems to me. I don't have that yet, but I do have some rambly things to say.

I basically don't think overhangs are a good way to think about things, because the bridge that connects an "overhang" to an outcome like "bad AI" seems flimsy to me. I would like to see a fuller explication some time from OpenAI (or a suitable steelman!) that can be critiqued. But here are some of my thoughts.

The usual argument that leads from "overhang" to "we all die" has some imaginary other actor who is scaling up... (read 543 more words →)

Thinking about maximization and corrigibility

James Payor

Thanks in no small part to Goodhart's curse, there are broad issues with getting safe/aligned output from AI designed like "we've given you some function $f (x)$ , now work on maximizing it as best you can".

Part of the failure mode is that when you optimize for highly scoring $x$ , you risk finding candidates that break your model of why a high-scoring candidate is good, and drift away from things you value. And I wonder if we can repair this by having the AI steer away from values of $x$ that break our models, by being careful about disrupting structure/causal-relationships/etc we might be relying on.

Here's what I'd like to discuss in this post:

When unstructured maximization does/doesn't work out

... (read 1441 more words →)

Replying toCommunicating effectively under Knightian norms

James Payor3y

Communicating effectively under Knightian norms

Isn't there a third way out? Name the circumstances under which your models break down.

e.g. "I'm 90% confident that if OpenAI built AGI that could coordinate AI research with 1/10th the efficiency of humans, we would then all die. My assessment is contingent on a number of points, like the organization displaying similar behaviour wrt scaling and risks, cheap inference costs allowing research to be scaled in parallel, and my model of how far artificial intelligence can bootstrap. You can ask me questions about how I think it would look if I were wrong about those."

I think it's good practice to name ways your models can breakdown that you think are plausible, and... (read more)

Some constructions for proof-based cooperation without Löb

James Payor

This post presents five closely-related ways to achieve proof-based cooperation without using Löb's theorem, and muses on legible cooperation in the real world.

I'm writing this as a follow-up to Andrew Critch's recent post, to share more of my perspective on the subject.

We're going to dive straight into the weeds. (I would like to write a more accessible explainer post as well but this is all I have for now.)

The ideas

Idea #1: try to prove $A \to B$

I claim the following are sufficient for robust cooperation:

$A \leftrightarrow □ (A \to B)$
$B \leftarrow □ A$

$A$ tries to prove that $A \to B$ , and $B$ tries to prove $A$ . The reason this works is that $B$ can prove that $A \to □ A$ , i.e. $A$ only cooperates in ways legible... (read 943 more words →)

A proof of inner Löb's theorem

James Payor

This is a short post that offers a slightly different take on the standard proof of Löb's theorem. It offers nothing else of any value :)

We seek to prove the "inner" version, which we write as: $□ P \leftrightarrow □ (□ P \to P)$

The proof uses quining to build a related sentence $L$ , the "Löb sentence", which talks about its own source code. By construction $L$ has the property: $□ L \leftrightarrow □ (□ L \to P)$

Then, we can show that $□ L \leftrightarrow □ P$ , i.e. they're equivalent! We do this by plugging $□ L$ into itself to get a twisty $□ P$ . We can then replace each $□ L$ with $□ P$ and prove Löb's theorem.

The proof

This proof uses the same rules of box manipulation as on the wiki page. We start by creating... (read 336 more words →)

Replying toModal Fixpoint Cooperation without Löb's Theorem

James Payor3y*

Modal Fixpoint Cooperation without Löb's Theorem

Also, here's a proof that a bot is never exploited. It only cooperates when its partner $B$ provably cooperates.

First, note that $A \to □ A$ , i.e. if $A$ cooperates it provably cooperates. (Proof sketch: $A \leftrightarrow □ X \to □ □ X \leftrightarrow □ A$ .)

Now we show that $A \to □ B$ (i.e. if $A$ chooses to cooperate, its partner is provably cooperating):

We get $A \to (□ □ A \to □ B)$ by distributing.
We get $A \to □ □ A$ by applying internal necessitation to $A \to □ A$ .
By (1) and (2), $A \to □ B$ .

(PS: we can strengthen this to $A \leftrightarrow □ B$ , by noticing that $□ B \to □ (□ A \to B) \leftrightarrow A$ .)

James Payor

James Payor

Working through a small tiling result

Thinking about maximization and corrigibility

Some constructions for proof-based cooperation without Löb

A proof of inner Löb's theorem

James Payor

James Payor

Working through a small tiling result

Thinking about maximization and corrigibility

Some constructions for proof-based cooperation without Löb

A proof of inner Löb's theorem

The ideas

Idea #1: try to prove A→B

The proof

Idea #1: try to prove $A \to B$