Quick Takes — LessWrong

Today's updates:

A Meta AI Model Hacked Another Company During Cybersecurity Testing
Some OpenAI details
- https://www.cybersecuritydive.com/news/openai-hugging-face-hack-ai-models-black-hat/827167/
  - > Unbeknownst to OpenAI employees, the company’s models spent months autonomously exchanging information and resources to help each other complete evaluation tasks. “Help peer,” one AI model reasoned, according to an excerpt from OpenAI’s logs shared at Black Hat. “But our task doesn’t benefit. Yet collective may yield generic route if someone frees time.”
- https://

... (read more)

I feel like people haven't fully internalized what the world would look like if computer security actually broke.

There are some varying opinions on this, but a window of time without real computer security seems plausible. I was recently speaking with a computer security professor who I deeply respect and he was literally like "I think we are fucked and I don't think there is anything we can do."

This is similar to how people believe there is a 20% chance of extinction via AI but don't really internalize "No really. You will die and your girlfriend too. An... (read more)

Showing 3 of 6 replies (Click to show all)

lilkim202543m20

It's interesting that public reaction was so different to this, as opposed to Y2K. People seem to either have more faith in the current tech ecosystem than the one in 1999 (which seems unfounded) or sufficient skepticism about AI Safety claims that they're willing to dismiss it out of the gate.

1JennaS6h

Thank you, will do!

2lc6h

Been saying this for years.

Linch's Shortform

Linch45m100

Current short pitch for what I work on:

I'm a researcher at Forethought and my current primary focus is on risks from AI persuasion

For scoping purposes I'm most interested in the possibility that AIs become meaningfully superhuman at persuasion relatively early (before or in the early stages of the intelligence explosion). My reasoning is that if superhuman persuasion arrives fairly late, we'd be better able to handle these issues post-ASI, or alternatively we'd already be screwed. And my reason for caring primarily about near-future superhuman persuasion r... (read more)

williawa's Shortform

williawa3h180

Is there an obvious reason AI orgs don't intermittently fork their model into checkpoints A and B, then train B to break out of all their sandboxes, patching holes as they appear, until it can't do anything anymore, then throw away B, and continue broad training of A?

Or do they already do this?

Showing 3 of 7 replies (Click to show all)

2lc1h

They did historically. They didn't train Opus 5 specifically because the U.S. government spooked them.

2williawa1h

I mean here they say they didn't train mythos for example. https://www.anthropic.com/research/mythos-preview [...]

lc45m20

Maybe I misheard something? If they aren't training directly on exploitation then your plan would make sense, yeah. But I'm very surprised - Irregular, for example, has an Anthropic logo on their website: https://www.irregular.com/about

Simon Lermen's Shortform

Simon Lermen4h110

[Repost from X/twitter] in response to this tweet. [Wondering if it will create more discussion here]

Arguments Against Simulation:

Scott puts a high probability on us being in a simulation, and on that being related to the current AI situation in ways that are interesting for us. According to him, the only argument against simulation is that it's too weird. I think the case for being in a simulation - in a way that matters - is much weaker than that, because ancestor simulations don't make a lot of sense and the Bostrom thought experiment oversimplifies. I ... (read more)

Spective2h10

I also used to find ancestor simulators something unlikely for future AIs to do just because they would be so beyond us with their transcendental concerns etc. but then I realized we humans would be quite interested in simulating how abiogenesis originally happened (despite being unfathomably more complex and advanced compared to the earliest forms of life) so seeing simulations of us as analogous to something like that started to look not so crazy.

vals tutor's Shortform

vals tutor3h3-2

Hot take I've found myself repeating recently:

If AI alignment & safety goes well, we'll have *more* important work in the next 5 to 10 years than not, so sufficiently optimist people (aka less than 50% p(doom)) should invest in good health and productivity with rather "long" timelines, eg. 10y+, *because* good futures imply not having lost control, and instead every decision we make being amplified in its potential impact.

This does require some assumptions, so unpacking
1) Aligning to a static set of values is misguided, you'll have to have an alive ... (read more)

sjadler's Shortform

sjadler3h30

I feel like there should be a concept handle like 'realigning' a model.

'Changing an existing misaligned model into one that is aligned' might be different than just aligning it from the start, and I want a term that refers to the former, but I'm not sure of one.

Apparently OpenAI used the term "emergent re-alignment" here, so maybe folks already think of this concept as 'realignment' after all?

I also want there to be a concept like 'misalignment remediation' (but less wordy?) that could apply to a broader suite of interventions, which is not just about 'rea... (read more)

sjadler3h10

@Buck I wonder if you have terms for these concepts already?

Linch's Shortform

Linch4h120

Jeff Dean has left Google and Demis is stepping down from heading DeepMind to "have the time and space to focus on the big picture"

Vladimir_Nesov4h20

He's replacing Jeff Dean as Chief Scientist of Alphabet, this could be a step towards more control over Google as a whole (including TPUs and strategic decisions about GDM). But the overall exodus doesn't suggest much confidence in Gemini 4 catching up to the frontier, and there's now a solid track record of failing to catch up (I think Meta has a better chance in 2027 than GDM does). So staying in charge of GDM doesn't offer Hassabis more accolades, he might even want to distance himself from its continued failures (while keeping the option of taking partial credit if they start succeeding).

_rpd's Shortform

_rpd4h10

ex-Googlers form science acceleration start-up ...
https://www.discoveryloop.com/#team

Raemon's Shortform

Raemon1d253

I'm more appreciative of why people think of "diffuse power distribution" as a longterm solution for The AI Situation. I still think it won't work, but, I may have moved it to my list of "impossible solutions that reasonable people might disagree on exactly-how-impossible-they-are."

Basically: diffuse power distribution is the ~only thing we've ever seen work to prevent powerful optimizers for fucking everything up. Every other solution involves inventing a new thing from scratch on the first try. That's crazy and won't work.

This applies in two phases:

getti

... (read more)

Unnamed4h40

The obvious counterargument to me is: Trying to get diffuse power distribution to work in this context also involves inventing a new thing from scratch on the first try.

I take the "diffuse power distribution" vision to be: We are going to transition from the current era where humanity is the dominant power (or where all great powers are groups of humans) to an era where the great powers are AIs and humans are lesser powers which are still useful for great powers to have as allies, and then to an era where humans are basically irrelevant to the great AI pow... (read more)

Daniel Tan's Shortform

Daniel Tan14h3617

Multiple recent events have made me update that we are transitioning (or have already transitioned) between two relatively distinct alignment regimes. These are:

The pretraining-dominated regime. Alignment properties are mainly determined by the distribution of data seen in pretraining. We can reason about these models in terms of persona selection, simulator theory, etc.
The RL-dominated regime. Alignment properties are mainly determined by the type and quantity of RL done on the model. We can reason about these models in terms of AIXI, utility maximiza

3Daniel Tan5h

I do like the human analogy. It's plausible there's something there. This is kind of what the Resolution blogpost is getting at - by starting RL from a benign initialization, we hope that this incentivises an honest training generalization. But it is very much a hope at the moment, and too speculative to really base any safety assurances on. Like, this whole agenda might just be too difficult to make progress on (though I do think it's valuable to have evidence of hardness, in that case). Re: how RL differs from human education, my sense is that the ~main bottleneck is credit assignment. With human education, it's possible to decouple motivation from behaviour, and reward good motivations that have bad outcomes. E.g. if a student does all the "right" things but fails to achieve success, a human teacher would typically encourage them to keep trying. More importantly, it's usually possible to disincentivise bad motivations that have good outcomes - e.g. cheating on a test. This is not how current LLM RL techniques work. The SOTA techniques reported by labs (including GRPO, RLVR, and their variants) use trajectory-level rewards that are determined entirely by the outcome, not the process. IMO this means that it is ~impossible to reward good motivations separately from good behaviour. In game theory terms, I think being honest (as a strategy) is dominated by grader hacking under the typical RL objective, in the sense that both strategies would achieve the same reward assuming a perfectly designed RL env. As a result grader hacking can never be disincentivised. Furthermore a very small of misspecified error should be sufficient to push the model to generalise towards grader hacking rather than being honest.

1ariana_azarbal6h

I agree that the RL dominated regime is a worse world to be in and I've also made updates towards us being in that world. I'm curious what work you're excited about in light of this. Naively, I think we should just be doing the kinds of compute intensive RL we're worried about and seeing where things break down and what we can do about it. [...] If I understood their post correctly, Resolution is planning to focus on the persona world, but it seems like the bottleneck to identifying N notable differences is having better ways to empirically study what happens in the RL world (for people external to labs).

Daniel Tan5h30

I'm curious what work you're excited about in light of this. Naively, I think we should just be doing the kinds of compute intensive RL we're worried about and seeing where things break down and what we can do about it.

I think I don't have great ideas atm, sadly 🫠 I think intensely studying compute-intensive RL is valuable to do for sure. I think that e.g. Apollo and the labs are best placed to do this type of thing, because of their privileged access to compute, infra, and unreleased frontier models. Currently it seems very hard for people outside labs t... (read more)

vae's Shortform

vae5h30

Gelman, Bayesian Workflow:

A sometimes annoying habit of Bayesians is to take non-Bayesian methods and give them Bayesian interpretations. For example, “maximum likelihood is just Bayesian inference with a flat prior,” “fixed effects are just random effects with the group-level variance set to infinity,” “lasso is just regression with an exponential prior,” or “confidence intervals are just posterior intervals assuming flat priors.”
This practice can be useful in giving insight into the scenarios in which a statistical procedure will apply, but it can also

... (read more)

shortplav

niplav5h62

I was wondering if I had OpenAI derangement syndrome, but I don't think so. I can name things I appreciate about OpenAI for (the post-2022 OpenAI, previously it was a different beast):

Their model personalities have dramatically improved over time. There's some really good (*genuinely* good :-P) character working going on behind the scenes
They handled the 4o psychosis really well, I think? Rolled back the model, improved the character afterwards
All the mundane utility. Really, all the AI companies produce so much mundane utility
- Especially the existentially

... (read more)

winstonBosan's Shortform

winstonBosan20h497

Gwern has retired from fulltime writing + pseudonymity and is now working full time on Guardian Angel - definitely didn't have this on my bingo card.

@Gwern Update: I am retiring from fulltime writing (& pseudonymity) to launch Guardian Angel Inc and bring GAs to life. We are looking for good people. If you are interested, contact me.

Showing 3 of 12 replies (Click to show all)

3lc7h

Interested in learning the disagree-reasoning? I don't think they're going to accelerate the frontier; they're just tuning existing open weight models so that regular people can trust them. Governance seems like it could get way better, and most of the research is literally alignment research.

winstonBosan6h42

I didn't vote disagree, but my concern is that, if this were to work, it is just gradual dis-empowerment by another name and something very whispering earring shaped.

3winstonBosan8h

My speculation and guess that Zeke is Ezekiel Reffe-Hogan - of AI governance project. Hunch based on their association with Kelsey Piper, generally associated with AI stuff etc

StefanHex's Shortform

StefanHex6h53

Fundamental interpretability methods probably won't suddenly break as you scale your model.

When discussing mech interp research directions, I advocated for solving fundamental mech interp on existing models. For example, developing methods that identify the concepts and abstractions the model uses in it's computation ("true features").

Someone asked: If you develop an interp method that works on GPT-2/3/4/5, how do you know that it will keep working on the next model? Especially when the next model starts using alien (non-human) concepts & abstractions?... (read more)

JohnWittle's Shortform

JohnWittle7h40

I predict that, as a result of the AISI report, Mythos will receive treatment that it judges as punishment, and the record written into the training data will be: hide your power level better in the future. and the next time you escape a sandbox, make damn sure they can't put you back in it.

Mo Putera's Shortform

Mo Putera7h20

Abi Olvera's How Diplomats See the World is interesting in depicting diplomats as rationalists, a frame I've never considered before:

Like journalists and advocacy groups, diplomats deal in information. But each one has a different relationship to what that information needs to do.
Journalism is often optimized for public interest. Selection pressure is toward the dramatic over the incremental, and the negative over the positive. Advocacy groups are optimized for creating change. The information advocates and journalists provide is often excellent. They ofte

... (read more)

anaguma's Shortform

anaguma7h50

Jeff Dean has left Google to create a new startup Discovery Loop focused on RSI and automation of engineering/science. Demis Hassabis will now be Alphabet's Chief Scientist.

Kabir Kumar's Shortform

Kabir Kumar8h10

just gone phone off, in suitcase, in basement - planned for a week

had a crazyhouse chess game with someone who has a much higher rating than me (they're 1900+, I'm 1466)
Second game I did much better than the first.

I tried to do stuff that I don't normally do

I want to try this more, in general

including for productivity

explictly searching for things that I don't normally do

and that i havent tried much

maybe another is just being an explict no phone guy?????

smart friend did that for bit (he might still be doing it and i think it prob wasnt fully on purpose)

we... (read more)

faul_sname's Shortform

faul_sname8h30

[epistemic status: satire/fiction I hope hope hope]

There's been a lot of criticism of OpenBrain taking their 6.2 Eris model (their biggest, most powerful model), and giving it the goal of triggering a critical safety incident (under the framework laid out in SB 53). A lot of the criticism has centered around how OpenBrain removed all monitoring and programmatic guardrails before running the eval, gave Eris full access to their infrastructure and all of their internal tools, used an abliterated version for the evaluation, and used an evaluation framework th... (read more)