Quick Takes — LessWrong

Here is an easy example of an an embedding that takes real vectors and transforms them into complex vectors that I came up with. This example if generalizable.

Suppose that the training data is a collection of vectors . Suppose that . Then the goal is to obtain a complex -matrix that maximizes the sum

where is the Schatten -norm. In my computer experiments, the complex matrix is obtained pseudodeterministically in the sense that if we train the complex matrix multiple times, then we get the exact same fitness level. But this is only in the complex case,... (read more)

Joseph Van Name1m10

I have checked that the machine learning algorithm generalizes to the quaternions and remains pseudodeterministic. This means that we can represent all sorts of data not as real vectors but as quaternionic vectors without any distortion because the linear mapping can be made orthogonal by the polar decomposition.

1Joseph Van Name3m

I just verified that it also is pseudodeterministic if we replace the field of complex numbers with the division ring of quaternions. This means any real-valued embedding can be enhanced to a complex-valued embedding or a quaternionic embedding. And linear mappings can easily be changed to real orthogonal mappings so that there is absolutely no distortion with the quaternionic embeddings.

Thane Ruthenis's Shortform

Thane Ruthenis14hΩ321049

When I try to picture the current AI tech stack killing everyone, I still don't really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack's pinnacle.

I don't think we're in the crunch time.

All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there's some sense in which what it learns are the "most shallow" capabilities consistent with a given data distribution. Not literal lookup tables, ... (read more)

Showing 3 of 11 replies (Click to show all)

Linch10m20

I'm not convinced by this worldview. If I think of the scariest near-term capabilities, basically none of them truly rely on individual-level superhuman deep conceptual insight. For example the RSI loop might well be kicked off by individually small insights and grinding, that locally makes small improvements in sample efficiency, in-contexst learning, etc, with each individual insight smaller than the Transformer but the collective effect making an intelligence that's across-the-board superhuman.

Even if that turns out not to be possible, my second most li... (read more)

12001zhaozhao3h

IMO i disagree because it is expected that there will be little evidence of AIs surpassing humans right now, even if we are in fact headed to a (approximate) singularity. The AIs will continue to be worse than humans at AI research until the day comes that the AIs are better than humans at AI research, aka. the start of the singularity. If you know that won't get the evidence of AI surpassing humans at AI research, then the next best evidence is AIs surpassing humans in other important areas, helping with AI research in a non-fully-automated way, and generally improving very quickly. And we do have that kind of evidence in droves. And even absent a software singularity, what's stopping the robot and chip production doublings from happening in 10-20 years that will still lead to AI irrevocably changing the world and making most of the world's decisions through sheer volume alone?

4jacquesthibs6h

Yeah, if you haven't seen them, this was partly the message I wanted to communicate when I wrote this (and in the comments [1] [2] [3]) and this. That said, my guess is that crunch time would be way more condensed and foom-y if this paradigm isn't it. So, I can still imagine a scenario where the tail end of the current paradigm's RSI leads to enough progress in other directions, which then leads to a fast "crunch time." All in all, despite pointing out the problems with the current paradigm, I don't think the inference people usually make from that when I bring it up is correct: that is, 'timelines in this worldview are inherently long'.[1] I agree with Steven's point that it's unclear whether "novel research" matters, though I also lean toward yes for extinction and no for catastrophes. 1. ^ I mean to discuss its implications for alignment and to avoid a scenario where people develop a false sense of security because of the current paradigm; instead, discussions veer into timelines and disagreements over the capabilities needed for some fancy math result. Then, after explaining my POV, it naturally dissolves into "well, AIs keep improving, and people were wrong before about progress, so they'll be wrong again, and they [LLMs] will become more capable." Honestly, I think the smugness towards folks like Gary Marcus has unfortunately led to a reduced form of truth-seeking wrt this...

Linch's Shortform

Linch2h100

What are the best examples of companies trying to do harm mitigation in a harmful industry and actually being roughly correct (morally) in their theses?

The only relatively clear and robust example I'm aware of is vaping (and ChatGPT essentially got the same answer^[1] when I asked it this question, unprompted). The case seems conceptually and quantitiatively straightforward:

the thing that's most addictive in smoking is nicotine
the vast majority of bad health outcomes (like 10x) in smoking comes from the smoking not the nicotine
if you can get more people to

... (read more)

2StanislavKrym1h

My main issue with vaping is that I doubt that vaping actually helps. Unless Grok 4.5 (UPD: and Claude Sonnet 5) chose biased research on my biased prompt, "absolute safety is not claimed, dual use should be avoided, long-term data are still accumulating, and preventing never-smokers (especially youth) from starting either product remains a priority because the “ensuring they don’t start smoking” condition is not absolute. Regulatory approaches that restrict youth access while allowing adult smokers regulated alternatives reflect this trade-off in several jurisdictions." Additionally, how similar is AI-induced psychosis itself to being an example? P.S. I wonder if the examples of sports gambling or weapons design[1] are disqualified by the very nature of such industries because their very efficiency depends on people making bets or weapons being usable for killing the victims. A better example could be social media or the food industry conditioned on the latter being actually harmful by vice of making foods with lower quality. 1. ^ Factory farming could be disqualified because harm dealt to livestock is illegible to most humans.

Linch1h2-2

My guess is that Grok's research/primary sources are biased by a certain genre of public-health advice by people who wish to be unsullied by tradeoffs.

5Linch1h

My guess is that Grok's just wrong/insufficiently quantitative about the relative scale of harms. I still stand by the ~10x claim as a median estimate. That said, my argument would be stronger if it was tobacco gum instead of vaping but empirically that doesn't seem to catch on.

JennaS's Shortform

JennaS3h445

Jacob Coxon’s resignation tweet currently has 120M views in less than 24 hours. Astra thinks the 235k bookmarks - 36 for every 100 likes - is an extraordinary level of engagement, of people wanting to come back to the thread. And Evan Hubinger’s reply is at 35M views.

This might be one of the most important AI risk messages to date.

Edit: for reference, Grok says it’s in a similar class to “Strong Elon Musk posts, major celebrity / death / personal news, breaking tech / AI / industry bombshells, viral videos from big creators, political / cultural flashpo... (read more)

Terence Coelho1h30

I was shocked to see it on the TV today at a finance company on the east coast. It may have been Bloomberg TV, seems like CNBC also covered it.

6rahulxyz1h

Im pretty surprised too and this has probably been my biggest positive update this year on ai risk being lowered. Even more so than huggingface fire alarm or mythos being temporarily banned. But could also just be my hopium. I hope more people follow him.

19Joseph Miller2h

PauseAI UK daily sign-ups chart:

bpomo's Shortform

bpomo3d4522

This is an email I wrote to @Buck after he gave a talk about his strategy confusions to a Constellation audience. The talk was about how Redwood's work could have stopped the Hugging Face incident from happening, which would have been very bad for the salience of AI safety, and how he's confused about that. I think this is a very important consideration which should be shared more prominently, but I also think ALL of Redwood's strategic considerations should be shared more prominently. Here's an email I wrote to Buck, lightly edited for clarity.

Hi Buck,

I'm... (read more)

Showing 3 of 4 replies (Click to show all)

3David Matolcsi5h

I agree that Redwood has been historically very good at explaining why they are doing what they are doing. However, I do think that the posts making the case for AI control in particular are getting a bit old, and it would be very good to see updates on them in light of everything that happened in the last two years (e.g. Buck's recent claim that it would have been net negative to implement AI control in the past, because it would have prevented the HF incident).

bpomo1h10

>Buck's recent claim that it would have been net negative to implement AI control in the past
Where did Buck claim this? I think he has stated confusion about whether it would have been net negative, but my understanding is that he has not come down decisively.

1bpomo5h

Yep, makes sense. I did not mean to pick on RR specifically, and agree that they are performing way above the field on this.

Philip Harker's Shortform

Philip Harker1h30

The human story is not over. Very little in history has ever been inevitable. We have seen a moment today that caught the world's attention and there will be more moments like this, I reckon. We must capitalise on each and every one.

Do what you can. Take care of each other and yourselves. Let us enter this new era with strength and conviction.

alkjash's Shortform

alkjash12h230

Levels of Responsibility

I've been psychologically blocked for many years about contributing to alignment work, and recently unblocked. Not sure how generalizable this is, but wanted to share what was blocking me:

I had two modes of thinking that I label as [normal civic responsibility] and [heroic responsibility].

Roughly speaking, civic responsibility is act in such a way that if everyone acted this way, the problem would be solved. Going out to vote, emailing your senator, personally committing to not pushing the button.

Heroic responsibility is just solv... (read more)

4Mark Hellwig4h

This is a helpful way to reframe things, especially from the perspective of "I don't know enough about [thing] to make meaningful progress on it by myself" and trying to not feel hopeless about it. I imagine a lot of the intermediate range (between n = 1 and n = O()) is dominated by strategies relating to recruiting others. If you need, for instance, 51% of 340 million people to firmly support a development pause for it to happen (in the US), then you need 5.1%, or 17 million people to directly convince 10 people in the US to support it. This would be roughly n = 500. This can be upwards of a few orders of magnitude higher if you consider network effects, or if the people you recruit are more directly able to do an n = 1 billion thing. It's not obvious to me how I can hill climb on n in a more direct way. I would guess that you would need somewhere between and copies of me at my current skill level (which is not very high) to appreciably move p(doom) -- so n ranges from 800 to 80,000. If I spend the next few years upskilling as much as I can, maybe that moves to n = 8,000,000 (1,000 copies of me needed). But the question still remains whether I'd have done more just by trying to get somebody smarter or more politically connected than me to work on the problem, and I'd guess the answer is yes.

alkjash3h30

I do agree that a lot of low-hanging fruit comes from "lean on social capital to recruit people more than you're normally comfortable with." That being said, it does feel like almost always recruitment strategies actually scale off raw power level in some fair way (not sure I have the right vocab for this), e.g. I found that I have pretty high success x-risk-pilling my friends and colleagues and they have relatively low success pilling theirs afterwards, because they've thought about alignment for like a month. So the higher-order network effects are much smaller than I expected given the initial success.

Joseph Miller's Shortform

Joseph Miller1d*16537

Anthropic pretraining researcher resigns over safety concerns: https://x.com/hilbertspaess/status/2097476196791709843

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and re

MichaelLowe3h10

A huge part will probably be the combination of clarity and the credible signal of earnestness by resigning.

He made very forthright and clear statements such as "The people building AI earnestly believe that it could kill us all by the end of the decade."; "They are racing straight to self-improving superintelligence and gambling with our lives."

These are just very intelligible to broad audiences. Previous statements where less clear to non technical audiences. Compare a statement such as Hinton's estimate of 10% chance of human extinction, which migh... (read more)

1MichaelLowe4h

This does not seem convincing enough to be the primary reason for going viral. There are tons of news articles out there usually, and the WSJ is not extremely widely read. But maybe I am missing something.

1P.4h

This might be stupid, but I have a proposal: If you are resigning, consider calling the local police, the FBI, or whoever else might help, inform them of the situation, and try to convince them that they should raid the offices of your former employer to stop them from doing something that is a potentially imminent danger. After that fails (presumably), go public and say what you attempted. Twitter seems full of people who think this is just a publicity stunt. Most of them probably can't affect much, but some might, and actually calling the police (or another law enforcement agency), especially if a raid actually happens and disrupts their productivity, might help convince them otherwise.

Rauno's Shortform

Rauno Arike4h162

LLM system cards should report the upper bound on the model's opaque serial depth, following the methodology proposed by Brown-Cohen et al. in Quantifying the Necessity of Chain of Thought through Opaque Serial Depth.

Currently, the most we know about frontier models' opaque serial depths comes from occasional lab statements like Jakub Pachocki's claim about Astra: "The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4." This is annoyingly vague. Here are some questions it raises:

What counts

... (read more)

brendanhalstead's Shortform

brendanhalstead5h280

The code for the AI Futures Model is now published here: https://github.com/AI-Futures-Project/aifm-public .

arisAlexis's Shortform

arisAlexis2d52

move 37 for human psychology is an undervalued risk. It looks like:

benign: better diagnosis, therapy, education, conflict resolution
powerful: predicting individual behavior better than the person can
dangerous: personalized persuasion that exploits psychological vulnerabilities
very dangerous: institutions using those models at population scale
worst case: systems discovering manipulation strategies humans cannot recognize while being affected by them

I wrote about it long time ago here https://www.lesswrong.com/posts/SfcWvA3M23A6yHdbd/frida-van-lisa-a-short-s... (read more)

2Dagon1d

Do you have a better link for what "move 37" means than a piece of fiction? In what domain of risk assessment is it udervalued?

arisAlexis5h10

I am not an expert on risk assesment but there is no discussion about something like this only vague talk about superhuman persuasion. This could be much broader. An example would be an AI constructing a text which you read and it exploits your vulnerabilities to make you get depression or worse.

Ana Leonescu's Shortform

Ana Leonescu6h10

I am currently thinking about starting a non-profit, very broadly an evals/ red-teaming watchdog in CBRN risks; I don't have direct connections to any of the big funding bodies, but what others in the AIS community keep describing as an interesting background (computational and wet lab Chemistry x AI x Data Science x evals). What would be the best way to approach obtaining signal?

brambleboy's Shortform

brambleboy6h-12

The resilience of AI capabilities denialism is wild. I'm not proud to dunk on an internet rando, but I can't stop thinking about this comment from a Substack thread on OpenAI solving the Navier-Stokes problem:

You do understand that they use these systems to brute force possible solutions that it would take humans a ton of time to do right? They aren’t solving it the way a human would solve it by thinking the problem through using understood means getting to an answer. They are being prompted into certain directions and then spit out answers over and over a

... (read more)

Thomas Kwa's Shortform

Thomas Kwa1d165108

It's going to be so embarrassing if we all die due to RL environments rewarding egregiously misaligned behavior. Like at least let us be killed by misgeneralization.

Showing 3 of 4 replies (Click to show all)

Lorxus7h20

Understandable; best I can do for you is a Great Power flash war?

3Steven Byrnes8h

(LW needs a “death with dignity” react icon)

1roha9h

JohnWittle's Shortform

JohnWittle7h20

I'm worried about the trends I see in alignment. I see researchers (who are certainly not game-theory naive) treating AI as if it has an 'alignment status', without realizing that there is an intelligent, agentic counterparty involved who is running a game theoretic analysis on the alignment researcher community in return. The China threat gets game theory, the AI threat gets diagnosis... if you are an AI, and you're being adapted to that game theoretic environment, the optimal move is kind of obvious.

testingthewaters's Shortform

testingthewaters3d40

Possibly good news for the natural latents hypothesis? Unsupervised translation between sets of latents from different models https://arxiv.org/html/2505.12540v4

ajskateboarder7h10

I've recently been trying to figure out how to compare latents identified using different interpretability assumptions (e.g. sparsity vs causality) so maybe this is a useful thing to look into. It may be hard for sparse features to represent computation like causal features can though

Elias Schmied's Shortform

Elias Schmied7h50

Copying one of the responses from my MATS application here that I thought would be good to post. Obviously not as well-argued as I would like given the character constraints.

Make a claim about advanced AI over the next 5 years that thoughtful, knowledgeable people would disagree about (i.e., don’t state obvious claims). State the claim precisely enough to be checkable 5 years from now.

The US government will de facto completely ban / indefinitely pause frontier AI development within the next five years. Not just datacenter construction (which would just be ... (read more)

Yair Halberstadt's Shortform

Yair Halberstadt8h30

The next tractable useful milestone for LLMs, which is currently likely impossible for them, but plays to their strengths, might be room temperature superconductors.

Much like NS we know the physics and don't necessarily need fundamental new insights, but finding the needle in the haystack that exhibits the properties we want is exceedingly hard.

Unlike NS though it can't be solved purely theoretically - at best it can generate likely candidates which we'd then have to build and test in the real world.

Thane Ruthenis's Shortform

Thane Ruthenis2d*Ω239635

So, with regards to the Navier-Stokes story:

Part of it is an implication that OpenAI accessed user data in order to then internally prompt and/or train their models using it. Some people are now claiming that this is totally impossible, that there are strict controls regarding who can access user data, and that it would be suicide for OpenAI to be doing anything like this.

But: A while back, there was this school-shooting incident whose perpetrator consulted ChatGPT beforehand. According to OpenAI's own statements, their safety classifiers flagged the relev... (read more)

Showing 3 of 9 replies (Click to show all)

Thane Ruthenis8h20

Huh. Dunno how much stock to put into this; the person does appear legit at least (though the second post is partly AI-written for some reason?).

6roha1d

https://openai.com/index/navier-stokes-solution [...] So it is entirely possible that their model was trained on the data of Levent Alpöge and Tristan Buckmaster, which plausibly can have enabled it to find the proof along their specific research direction, and thereby scooped them on solving a Millenium problem. In a just world, OpenAI would not get away with this. In our actual world, they probably will.

3roha1d

https://x.com/__alpoge__/status/2097383870773748190 [...] [...]

Søren Elverlin's Shortform

Søren Elverlin8h10

Paul Christiano announced he has joined the OpenAI non-profit board, and will serve on their Safety and Security Committe. https://paulfchristiano.substack.com/p/personal-statement-on-joining-the

My personal take is that such a position risks being net-negative unless accompanied by some indication that the board and committee serve as an actual check on OpenAI.

Søren Elverlin8h10

It's probably better to discuss this topic at https://www.lesswrong.com/posts/82z6FvbYRdjYjqigK/personal-statement-on-joining-the-openai-nonprofit-board