Quick Takes — LessWrong

ControlAI has launched an official campaign in Canada, with 33 politicians across party lines backing a clear statement in favor of an international prohibition against developing superintelligent AI: https://controlai.com/canada-statement/en

I used to be ambivalent about government intervention wrt AI safety, because politics can screw things up. However, it seems clearly positive to me now: the development of superintelligence impacts the whole human species, and as such, the democratic process should have a say in whether (and how) superintelligence is ... (read more)

Showing 3 of 16 replies (Click to show all)

-3RobertM4h

https://x.com/ohabryka/status/2046703115551002808 You were in a related tweet thread that same day, responding to a tweet with the same basic claim in it. This isn't a vague accusation. I was literally in the room when Ruby got this request from Gabe.

Alex Amadori17m87

There is a recurring pattern of motte and bailey around these accusations. It always starts with accusations of very egregious high-level things like: "trying to get people to be coy around x-risks" or "being quite consistently deceptive and engaged in strategic conflation of AI existential risk with other issues to win political battles".

This is always in the higher level threads or comments, which more people will see. Taken by themselves, these are ridiculous claims, as is shown by Andrea, Gabe and Connor’s track record.

When pushed on this, habryka retr... (read more)

2JennaS9h

Would you be willing to give some examples of the things you have heard since the 2023 dialogue? I believed in ControlAI enough to apply for a position there, so this feels important to me.

Chamod Kalupahana's Shortform

Chamod Kalupahana1h10

Great post on using measuring logit distribution for eval awareness. Surprisingly effective, even for unverbalised eval awareness! ✍️

https://www.lesswrong.com/posts/PK7ZvFZxrgpYtrpF4/logits-as-a-new-monitor-for-evaluation-awareness-1

Chamod Kalupahana's Shortform

Chamod Kalupahana1h10

saw an interesting post for those also trying to transition into technical AI safety :)

https://www.lesswrong.com/posts/xiTBpBDwubnr4MLRe/trees-are-mostly-made-of-air-and-a-generalizable-lesson-for

Kongo Landwalker's Shortform

Kongo Landwalker2d-1-3

In my personal experience the peak of AI was exactly a year ago. While others were discussing hallucinations and spiraling, my chats could stay useful for hundreds of messages and allow me to work with huge tasks. Back then ChatGPT could even understand a new mathematical notion I came up with and engage in discussion of the theorems.

But during the last year i was mentally tracking how long does a chat last before it starts hallucinating (saying nonsense, or generating the human part of the conversation, or starting to repeat itself, or to ignore recent co... (read more)

2Brendan Long15h

You're stacking two problems here: Using the free models, and using two different models in the same conversation (which will confuse the new model since it's continuing a conversation it didn't write). Pay for one of the $20/month plans and use the top models and you'll see much better results. I keep conversations with Opus 4.8 going through multiple compactions with no noticeable problems, and it's very good at catching bugs[1]. [...] Google Translate is an LLM. 1. ^ I actually mentioned this on a recent coding-agent-allowed interview, that the bugs they slipped into the starting code aren't a good signal anymore since Claude caught all of them when prompted to "review this code before we get started". Some of the bugs were pretty subtle too -- the code worked and did something close to what it was supposed to do, and would only be obvious to a domain expert.

Kongo Landwalker2h10

0) my post was observation about personal experience with free tools, not about state of the art. They do not match. Maybe some of my speech patterns were more alligned with old architectures, keeping them sane, but drive new models crazy. Maybe new models are just worse trained in tasks i work with. Maybe i am just incredibly unlucky this year having to dismiss 60% of chats as hallucinating.

I find the state of the free ai more important, since the majority of humanity will never buy any model, so free models will have more impact on society (in an optimis... (read more)

xpostah's Shortform

samuelshadrach2h10

I am looking for a cofounder to cyberattack the US govt and US AI companies, and leak information about them

I will pay you $1000 if you refer me to a potential cofounder.

Signal: samueldashadrach.02

PGP-encrypted email also available on my website.

I am also open to some initial feedback, although I might take some weeks before properly responding to it. I am not actively in feedback-collecting mode right now.

The document is incomplete. However, I don't plan to complete it any time soon, so I figured it is better to post an incomplete document than to post no... (read more)

reinthal's Shortform

reinthal3h10

RePe Linear probes should not work when you have 200 samples and activation space is ten or hundred times that but they do and I don't know why.

leogao's Shortform

leogao1d50

a corollary to the hazards of arguing against bad takes: please don't write things that are defined entirely by trying to avoid the reader coming away with specific bad takes or misunderstandings people often have.

you should write things primarily to nail down the concepts unambiguously for an audience of generic smart people. your idea should be defined by what it is, and not what it is not. it isn't SCP-055.

if you really need to, add a "things i don't mean" section to concretely describe and disavow some common misunderstandings. but it should be possibl... (read more)

5leogao1d

relatedly: please don't let stupid and/or malicious people dictate your vibe. a common pattern i see is people being scared of saying anything that might sound optimistic, because someone will see it and think "oh, great, alignment is solved, thank goodness, i can keep accelerating capabilities / not doing anything to make things go well". it's very easy to make clear to all sane people that you think things will not go well. anyone who is still confused is either stupid, in which case their opinion doesn't matter; or looking for some excuse to do the thing they wanted to do anyways, in which case you are not counterfactual in them doing whatever; or malicious, in which case they are trolling you and finding it amusing to mess with you, in which case you shouldn't negotiate with terrorists. if it's too difficult to perceive this pattern in yourself, you can observe some of the climate change people and how they are often extremely upset by any suggestion that e.g solar is on track to solve big chunks of global warming. people will often refuse to engage on the object level question of whether solar really is enough, and say outright that if we adopt the attitude that climate change is on track then everyone will stop doing the things that need to be done, thereby throwing us off track.

Viliam4h30

I should put a reminder like this on top of my computer screen.

One reason I am often writing long comments is a feeling of defensiveness, as if I don't make my case perfectly ambiguous and bulletproof, by adding more and more words, of course someone will pick up the worst possible misinterpretation. (I had people like that in my life in the past.)

Ryan Meservey's Shortform

Ryan Meservey1d*40

Here are some strange things parents often believe about children and food:

If I do not make my child eat/drink, they will starve/perish.
Children's diets must be meticulously controlled or they will eat all the "wrong" things and too few of the "right" things.
Children are inherently picky and should be served separate "child-friendly" food.
A good way to incentivize bites of the "right" things is to make it a requirement before bites of the "wrong" thing.

My wife and I reject all four of these beliefs, and (n=2) our children are healthy, happy, and not at all... (read more)

Viliam4h20

I have two kids. One of them is happy to eat a wide range of meals. The other would prefer to only eat pasta and sweets, that probably wouldn't end well.

4Brendan Long17h

When I hear anecdotes like this, I always wonder how much of it is your behavior and how much is genetics. If your kids were demanding nothing but mac and cheese would you be as happy about this setup? A friend of mine has a kid who gets vitamin deficiencies when allowed to select their own food (literally 100% Kraft mac and cheese).

3Ryan Meservey12h

You are wondering about the right things. I would say that there are a few more behavioral things we do that stack the deck in favor of vegetables for the kids choices: (1) we like to talk about the benefits of vegetables ("This will help you poop! There is an ongoing kid-constipation crisis."), (2) a little branding ("check out these dinosaur leaves" / "I bet our pet mice would love these"), and (3) we learned how to make delicious vegetables. My guess right now is that #3 is an underrated skill and more people would stop associating "healthy" with "unappetizing" if they new how to do it. As for what I would do if the kids demanded nothing but mac n cheese: ¯\_(ツ)_/¯. On nights we are not having Mac n cheese (most nights), they would need to eat something else we are having or be hungry. The way our system works, we don't generally try not to make special exceptions in the moment. (Edited to this more accurate statement; I am not completely immune to the requests of my kids and sometimes they are reasonable requests that make me a jerk to refuse. For example, my oldest wanted his tuna salad served on bread last night instead of rice, and I decided I was being dumb insisting on rice). I'm having trouble engaging fully with the hypo because some mix of genetics + behavior has given me non-picky kids. For the hypo to really bite, I would have to assume this isn't the case and the behavioral interventions have basically failed, in which case I guess I wouldn't endorse the system. Also, I won't claim that my system is any good for helping kids who are already picky—I don't know what strategies are best to help diversify already constrained diets.

leogao's Shortform

leogao1d3-6

theory: most people fall into one of the following categories (or some mix of them):

altruistic because they are motivated by empathy ("the warm fuzzies"). however, they are not inherently scope sensitive.
selfish but ashamed of this, so they develop an ideology where being selfish is actually good for others, Ayn Rand style, so no actions need to change.
altruistic because they feel shame about selfishness, so they develop an ideology where making some specific sacrifices cleanses one of sin. EAs are like this.
selfish but afraid of confronting this, so they

Viliam4h21

Not sure if there are separate categories.

One thing not in your list: enlightened selfish people who realize that promoting altruism as a social norm is a form of insurance, in case they also would benefit from receiving help one day.

2niplav19h

I like to make the distinction of altruism-as-social-rewards-hooking-into percepts and altruism-as-social-rewards-hooking-into-worldmodel. One can get warm fuzzies in either, would be my guess? Like, I'm EA, and I've also taken some unusually selfish actions over the years.

2leogao1d

if you mean veil of ignorance style reasoning, that's 3, unless they manage to so deeply galaxy brain themselves that they genuinely dissolve a normal sense of self and truly start being an open individualist or something. then, uh, i don't really know how to categorize them

Eric Neyman's Shortform

Eric Neyman19h213

Getting elected involves compromising your values.

Usually, when people say "compromising values", it carries a connotation of low integrity. That's not really my intention here. Instead, I mean it in a more neutral way: if you're running for office, it's pretty likely that your values will be out of step with your constituents' values in one way or another. Maybe you have a wider moral circle of concern, or have a stronger sense of justice, or whatever, leading to you holding views that are unpopular among your constituents.

This makes you less likely to wi... (read more)

Showing 3 of 6 replies (Click to show all)

leogao4h20

a much more extreme example of num 3 is LBJ. he spent decades acting extremely racist to get the support of the south, and then did a complete about-face in the presidency, utterly betraying his southern supporters and working to pass the 1964 civil rights bill.

2Unnamed8h

I'm surprised to hear that this is your take. Do you only feel this way about politicians who explicitly state that they are acting on their constituents' values rather than their own, and are willing to also talk candidly about their own values which they are choosing to set aside? Or do you feel similarly about politicians who consistently speak from the values that they have chosen to act on, and do not clarify that those are values that they have taken up on behalf of their constituents while their own personal values differ in some respects?

3habryka6h

I am not sure what the question is. I am not presenting particularly specific feelings about politicians here. Of the two things you list, the former seems better than the latter. The latter seems kind of fake and not how I expect things to actually work? I don't think politicians can turn themselves into perfect avatars of their constituents' preferences, so pretending you are seems bad form to me. At the very least there are indexical preferences that make that kind of fundamentally infeasible "I do not want to do this because it is very stressful" does not even quite compute if you try to purely model yourself as the perfect avatar of your constituents' preferences. Also, I think in everyday political work, you will just receive appointments that call for different levels of discretion. Being appointed to an advisory board usually calls for using more of your own judgement and models, whereas being appointed to a lawmaker position calls for channeling more of your constituent's preferences, so I don't think there is any high-integrity way to act if you can't model those two as separate.

Cole Wyeth's Shortform

Cole Wyeth2d5943

Over the last month or two I have rarely found it worth finishing any posts on the front page (even when I have done so, I usually felt ambivalent about the choice in retrospect). I am not sure whether I am simply busier with other things or whether the quality of the front page has went down, but as an incredibly avid lesswrong reader, it seems worth it for me to flag this bare fact (without proposing any particular explanation).

Showing 3 of 18 replies (Click to show all)

J Bostock4h20

Posts seem overly long at the moment. They get boring because the substance tapers out within the first 50%. Not sure why.

My first hypothesis is that this is an artefact of Claude 4.7+ being used for editing i.e. maybe Claude 4.7+ is a decent editor but prefers overly long essays. But I have literally no evidence as I don't use AI for editing passes.

6Ben Pace6h

I’d want to see any statistic before I believed that. There weren’t that many writers even writing on LW during Inkhaven 2, and the ones that were (e.g. Lawrence Chan, Katha Grace, Oliver Habryka, etc) I think mostly wrote pretty good stuff.

6Cole Wyeth15h

It could be, but I would hope that we could continue to build novel ideas here.

Bruce Middleton's Shortform

Bruce Middleton21h-40

ReMVIR: the discipline of sensitive engagement, focused by theorems (whether so named or not), which makes communities superconductive of knowledge.

The acronym stands for Resolution by Mutual Verified Iterative Ratiocommunation. Ratiocommunation is understanding perception and its rationale well enough that the agent will acknowledge your expression of them as sound. Iteration zooms in, increasing resolution at the point of divergence until it is clearly imaged.

Operative in many areas; a central activity of real science. The procedure of mutual mind-debugg... (read more)

Viliam4h20

Congratulation on inventing a word "ratiocommunation" that Google search never heard about. It is a rare achievement.

Apologies if there is a substance behind what you wrote, but to me it mostly sounds like word salad (and possibly AI psychosis). Seems like you are pointing at a concept of "one person understands what the other means" but for some reason you invent complicated words and theorems and proofs.

If you are using an AI, maybe ask it to explain the entire thing using words that a 5 years old would understand?

Arjun Panickssery's Shortform

Arjun Panickssery11h357

Bertrand Russell's parents died by the time he was four years old and in 1876 he went to live with his grandfather, the last Whig prime minister, who as a young man had met Napoleon at Elba.

In 1966 at the age of 94 he met Paul McCartney and converted him to his anti-war stance on Vietnam.

Previously, I had figured that this lifespan (roughly 1870 to 1960) was the most extreme length of history for someone to live through. You have to be born early enough to remember a time when big cities didn’t have street lights or automobiles, while ideally living to see... (read more)

Joey KL6h00

AGI will not just keep the show going, it will bring it to its grand finale.

12Maxwell Peterson10h

This is neither here nor there but: Syngman Rhee is from long enough ago that the romanization was nearly unrecognizable to me as a Korean name! I thought for a second that Korea must have had a president from a foreign country. (The modern romanization would be Lee Seungman.)

Rattengift's Shortform

Rattengift9h3-2

I have been revisiting the 2023 post "AI Timelines" today. I would be interested in seeing what the participants within would offer as estimates at the present time.

From my own cursory estimation, it seems like Kokotajlo was wildly miscalibrated on many or most fronts, while Cotra and Erdil's predictions in 2023 seem fairly calibrated, leaning heavily in favor of Erdil (who seems in retrospect to be remarkably well-calibrated).

Scattershot takeaways:

Kokotajlo correctly assumed governments would completely fail to slow down timelines.

Erdil's predictions are ... (read more)

StanislavKrym6h20

@Daniel Kokotajlo's most recent views are expressed in Q1 2026 Timelines Update. Maybe he will release a new update?

Edited to add: Why do you believe that the predictions of Cotra and Erdil are mostly correct? Erdil's prediction which struck me was the following:

Erdil's misprediction

My median world looks something like this: we keep scaling compute until we hit training runs at a size of 1e28 to 1e30 FLOP in maybe 5 to 10 years, and after that scaling becomes increasingly difficult because of us running up against supply constraints. Software progress cont

... (read more)

Orpheus16's Shortform

Orpheus1618h457

Anthropic releases new post about RSI and slowdowns-- also commits to doing verification research and verification-focused policymaker engagement. (In my view, this is probably the best post/announcement about AI risks that has been produced by any frontier AI company in recent memory).

https://www.anthropic.com/institute/recursive-self-improvement

We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.

Daniel Kokotajlo8h163

The political winds are shifting; the public is becoming very anti-AI; x-risk is becoming more in the Overton Window. The companies are therefore moving somewhat to avoid looking too bad to too many people.

19MichaelDickens13h

I put approximately zero stock in "commitments" by Anthropic, given their track record of how they use that word. But the fact that they want to conduct research on how to implement a pause is a promising sign.

2neo13h

I've always had an inkling that frontier labs would make increasingly rigorous safety communications as the threat became more salient. After all, most lab leaders seem to have recognized x-risk way before the AI boom and are presumably interested in continuing to exist. And in general, I think a lot of the safety community tends to extrapolate from current lab safety attitudes and not price-in changes to communications and concerns that seem likely as capabilities, and accompanying unease, grow. But I agree, this doesn't feel satisfying for some reason. Was it necessary to downplay/gloss over safety and scare the shit out of everyone concerned about this sort of thing? And if it was, for political or financial reasons, it means there exist incentive structures working against transparency of knowledge/belief, which seems bad overall and for the long-run. Side note - your link requires a login, not sure if that's intended.

Yeonwoo Kim's Shortform

Salmonus_Kim6d31

In the recent Gemini update, the ability to clear chat history all at once has been removed. Now, one must navigate to the settings, select 'Delete Activity,' and go through several additional steps, which I find rather frustrating. Furthermore, I have strong doubts as to whether deleting this data actually removes it entirely from Google's databases. I suppose I could simply stop using Gemini, but since this is more of a minor inconvenience than a critical issue, abandoning the platform altogether seems excessive. What are your thoughts on this matter?

RHollerith8h20

openrouter.ai accepts cryptocurrency as payment.

There's also "remote attestation", which Apple is investing heavily in for its AI cloud. "Remote attestation" is a combination of hardware features and cryptography meant to assure the public (well, technologists who have the time to review Apple's architecture) that the software running on Apple's servers is really what Apple says it is. The public can then review the source code for this software (which Apple plans to publish) to verify that Apple is not saving chat history.

Apple's scheme is only for devel... (read more)

3Viliam5d

This is a modern tech definition of "consent": we made it technically possible, but very inconvenient, to say "no". If the user does not have sufficient skills or patience to jump through all the hoops, we understand it as "yes". The equivalent in offline world is how easy it is to sign up to a service, and how difficult it is sometimes to cancel it. A predictable outcome of the following forces: * the company really really wants to get a "yes" out of you * the law requires the possibility "no", but specifies no details * the citizens are told that updating the law to requiring making "no" as easy as "yes" would be socialism The standard name for this kind of behavior "malicious compliance".

Eye You's Shortform

Eye You13h190

How do LLMs and humans compare with regards to the amount of energy, data, and compute that's used to train/run them? I was inspired by Samuel Knoche's post on sample efficiency to come up with some numbers. This table was made by Opus 4.8. after some iteration:

"Task" = "produce one thoughtful ~1,000-token answer"; unclear if this is a useful number, obliviously it doesn't generalize. I do think it's interested to compare the energy/computer ratio for inference between human and AI.

Training data for humans is a big "???". There's ~4x10^8 waking hours. Opu... (read more)

Andrii Vasylenko9h10

Some Fermi estimates:

On the lower end, 4 words/second times 16 bits/word times 4x10^8 seconds adds up to to ~3x10^10.
On the higher end, assuming 10^7 bits/second (about the retina-to-brain bandwidth), it adds up to ~4x10^15.

However, I think the brain is much less data- and compute-efficient than an optimal AGI algorithm would be. So I don't think it is a good predictor of how much data future AI algorithms will require.

Pee Doom's Shortform Feed

Pee Doom9h10

I've been interested in the potential of zero-knowledge proof type things to verify that computers are not running unfriendly AIs, to get the minimal amount of omniveillance that may be necessary to thread the Scylla and Charybdis of x-risk via extinction and x-risk via totalitarian stagnation. Each computer attesting that it's not doing <bad things>, with no more than a yes/no. Possible issues: maybe it's computationally infeasible, hard to operationalize, or can be used to do more intrusive surveillance. I know Drexler was interested in this. IIRC ... (read more)

KAP's Shortform

KAP12h3-2

Rather than argue for or against the consciousness of LLMs per se, I argue that consciousness in LLMs (or 'qualia') implies philosophical monism.

My claim is a logical implication, and not about the fact of the matter. If the implication holds, then:

If monism is false, LLMs are not conscious.
If monism is true, LLMs may or may not be conscious.

Because LLMs are not biological, if LLMs are conscious, conscious experience must be substrate independent. Nonetheless, biological brains and LLMs share the same substrate in the sense that they are both constr... (read more)

JBlack10h20

There are plenty of non-monist theories in which LLMs could be conscious, so the first bullet point is simply false. The second point seems true, but the combination just means:

Regardless of whether monism is true or false, LLMs may or may not be conscious.

Alexei's Shortform

Alexei1d-1-1

I’d make it highly recommend / mandatory on LessWrong for all posts at the top to disclose to what extent they have been generated using ideas from AI. I think a rule like that is already in place if the post is partially AI written. And I think knowing where the ideas are coming from is just as important. I’m beginning to suspect that more and more posts (everywhere) are influenced in large part by AI. In which case, why am I even here?

Ben Pace12h20

I do really think there's a lot of using-AI and a dearth of understanding-how-people-are-using-AI.

I think using AI is often good, but I would like more understanding of what's going on.

2niplav19h

There are the accounts Claude+ and chatgptopenai, which one could make co-authors on posts to indicate idea/feedback/whatever-involvement. In general, I'm happy to support developing norms about AI involvement, as to give people an out if they're worried about memetic plagues.