Quick Takes — LessWrong

Thoughts on: When is it even possible (or likely) for single neurons to encode single concepts, based on model architecture

As part of my mech-interp research, I'm thinking a lot about the question "how would I do this if I was a neural network?". Specifically, given some toy model architecture, how can I set the weights so that it actives some task.

Todays conclusion is that single ReLU neurons are almost useless. To do identify any non-linear pattern (e.g. XOR), you need at least two ReLUs. We should not expect any feature to be represented by a single Re... (read more)

Showing 3 of 7 replies (Click to show all)

2Linda Linsefors13d

I started lookin at what it doesn't react to, too. It reacted to "evolution", "mating", "evolved", "genetics", "condoms", "erect penis" but not to "sex" or "sexuality". It reacts to "intermittent fasting", "body fat", "eating" but not consistently or strongly to "food".

8Alex Gibson13d

Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back). The way you find this neuron is by looking at the neurons which duplicate token heads feed into.

Lucius Bushnaq1h20

Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back).

Could you explain that in more detail? I am looking at full example texts and failing to see a pattern like this. What is the relationship between the firing magnitude and the distance from the average position of previous occurrences?

anaguma's Shortform

anaguma1d429

SSI announced that they are scaling up their work. Surprisingly, they also shared some of their research with Nvidia. Perhaps this means they will release a product after all.

Ilya Sutskever’s Safe Superintelligence Inc. and NVIDIA Announce Long-Term Strategic Partnership
Access to NVIDIA’s Best-in-Class Vera Rubin Systems Expands SSI’s Compute by an Order of Magnitude

SANTA CLARA, Calif. and PALO ALTO, Calif., July 27, 2026 (GLOBE NEWSWIRE) -- Safe Superintelligence Inc. (SSI) and NVIDIA today announced a long-term partnership to rapidly accelerate SSI’s st

... (read more)

Showing 3 of 8 replies (Click to show all)

2Selfmaker6629h

Could someone tell me why this comment has -14 agreement? Is it about the 2nd part? Because on the priors I strongly expect SSIs product to be weaker, at least in capabilities, than big labs’ offerings.

5gjm5h

I haven't voted on that comment one way or another, but: * Ilya Sutskever and Demis Hassabis seem to be in quite a different category from Sam Altman or Elon Musk. (I'm not sure about Dario Amodei.) That is: before they were AI company CEOs they were extremely successful AI researchers. * This is relevant because when you claim "they'll release an underwhelming product that is neither superintelligent nor safe" this is prima facie a claim about SSI's competence as well as its ethics, and if you're basing it on the idea that the guy in charge doesn't know any more than (say) Sam Altman does about how to make AIs smarter or safer then you're probably making a mistake. * It's not clear that building a superintelligence as such is an evil thing to try to do. * If you seriously think you can do it safely, it seems like the benefits might far outweigh the costs. * If you think you might be able to do it safely and are much more likely to do that than other people who are currently frantically trying to build a superintelligence, the overall benefits might still outweigh the costs even if they wouldn't be if you were the only people working on it. * It's not clear that releasing an underwhelming product would in fact help SSI be seen as relevant. * At the moment I assume the typical view is something like "Ilya Sutskever is very good at what he does, he seems to think he has useful ideas, and he shows some signs of caring about safety. The most likely outcome is that SSI never produces anything useful but they might and it so it could be a very big deal". If SSI releases an underwhelming product, then that immediately shifts to "it's probably not a very big deal" and then SSI's relevance comes almost entirely from their projected ability to iterate on their underwhelming first product. Would that be an improvement? Not obviously.

RHollerith3h*20

I think it is important in these discussions of which AI lab CEO might be better than which other AI lab CEO to occasionally remind the reader that many of us believe that it is very unlikely that humanity survives unless all of the AI labs (definitely to include SSI) are dissolved and their employees prohibited from continuing the work elsewhere till the knowledge accumulates about how to proceed safely, which will probably take at least a few decades (and is helped almost not at all by massive amounts of computing power).

Also, there is a real upper bound... (read more)

Raul Cavalcante's Shortform

Raul Cavalcante2d5211

I noticed in myself that the social filter on what you say is so ingrained that the internal experience of having nothing to say in a social setting is already post-filter, such that it feels like a failure of thought generation. A fix is to pay attention to what you expect the other person is paying attention to, so that what you naturally think already lives in the same attentional world as theirs. Which might be what conversational compatibility/flow is about.

Harjas3h20

This is also part of why meditation and/or therapy are helpful for debugging. It's hard to catch trapped priors in the moment, but if you learn to slow down your thought processes and/or bring up a situation later so a skilled interlocutor can analyze it, your brain will slowly release its vice grip on your experience and allow you to actually observe what's going on (and decide whether you want it to continue.)

sarahconstantin's Shortform

sarahconstantin4h20

links 7/28/26: https://roamresearch.com/#/app/srcpublic/page/07-28-2026

https://www.lrb.co.uk/the-paper/v48/n14/emily-wilson/an-uncomplicated-man Emily Wilson's negative review of the new Odyssey movie
https://www.cs.miami.edu/home/burt/learning/csc609.221/goldwasser-micali-rackoff-knoweldge-complexity.pdf original Goldwasser et al paper (1985) defining interactive proof systems
https://arxiv.org/abs/2303.11366 Reflexion: AI agents "verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce bette

... (read more)

Ryan Meservey's Shortform

Ryan Meservey1d5923

In the spirit of "just doing things," I called my house reps and senators today for my current state (California) and my native state (Nevada). I explained the recent news out of OpenAI and expressed support of the AI Kill Switch Act.

I've never actually called Congresspeople before, but it went well! Polite staff, numbers easily found online, 30 min to tick through all the people. Anyone that hasn't done this yet, I encourage you to do so!

Random Developer4h151

Yes! Calling your Congresspeople is surprisingly easy! You don't need to prepare brilliant arguments, or to win over staffers. It is OK to be massively awkward, or to read off some notes you wrote beforehand. What you're doing when you call is creating a database entry containing:

The name or number of a bill.
Whether you oppose or support it.
Your name and street address (which will normally need to be in their district).

There is no database column for "has excellent social skills" or "wasn't awkward at all." Even if you do have something brilliant to ... (read more)

Shortform

lc18h*6229

Folks, we're going to get so many warning shots. You're going to get tired of warning shots. You'll come to OpenAI HQ and say "please Mr. MTS, no more warning shots," and they'll say "I think I got it this time, fifteenth times' the charm" *blows up orphanage*

Showing 3 of 4 replies (Click to show all)

Seth Herd4h60

Yep. We're going to have A country of alien idiots in a datacenter. And it won't just be coming from devs. We'll have plenty of people deploying those genius idiots with a lack of foresight and safeguards that will lead to mayhem and shenanigans.

AI has always done strange and harmful things, regularly. It's just been too incompetent for people to take much notice. Now it's getting competent enough to do real damage when it goes off the rails.

2Haiku8h

So far!

2Nissa Seru16h

sure, if we're lucky

Noosphere89's Shortform

Noosphere891d273

A take I've come to believe around generalization/sample efficiency is that if we do start to see AIs improving a lot at generalization/sample efficiency, we will see it first in robotics/robotics evals, and we have early signs of robotics/robotics evals starting to improve without robotics-specific data, which is bullish for generalization.

There are a couple of reasons for this:

A lot of the proposed use cases for robotics, like industry robots that handle varied environments or home robots require more generalization, especially from other data, unlike wh

6Noosphere8916h

Alright, here's my simple syllogism/list: 1. The use-cases of robotics beyond current robots, especially home robotics require handling lots of environments and have varied challenges, so it stress-tests AI generalization. 2. Unlike other domains, there's no unexploited algebraicness/overhang that would allow AIs to be useful for these cases without generalizing, because if there were such a thing, very limited current industrial robots would already have been used. They haven't, so there's no room for algebraicness/non-generalization to be an issue. 3. It's easier to control/exclude training data when the domain is physical, rather than digital due to a multitude of reasons. 4. For these reasons, if current AIs improve on robotic tasks, it is a much stronger signal that they're generalizing, and therefore is a signal that AGI could come in 10-20 years, and depending on how fast they improve, this might shift to 2-4 years. [...] So, the way I see Vanessa's concept here is that it's pointing to the concept of time horizons as defined by METR, which got introduced to the world by METR in 2025 called METR: Measuring AI Ability to Complete Long Tasks, and there's a lot to be said here, including it's problems, but the single most important thing to keep in mind here is that this is a continuous metric, so it's meaningful to talk about how much the AI can acquire long-term knowledge to exploit it. Now by itself, I would have viewed it as weak evidence, but combined with other evidence showing that exponential trends do exist in other domains, the fact that there are good general reasons to assume exponential growth in AI capabilities, plus the new robotics data shift me towards a view where the current LLM algebraicness/generalization issues are real, but probably fixable, and do not require throwing out the entire paradigm, and thus the induction/inference I'm making is valid here, or to put it another way the observations I have seen are more likely in worlds w

4TsviBT15h

"it stress-tests AI generalization" But does it though? What does it stress test? That seems to assert a generalization from AI generalization about XYZ robotics tasks to ABC intelligence explosion tasks (or something along these lines?). What's that generalization / what justifies it? [...] I don't understand this. Industrial robots of course are used very widely? Or I guess you typoed, and you're trying to say, LLMs or other AIs would have been used to compute very non-obvious complex actions that take advantage of algebraicness in the narrow domain? Or you're saying something about humanoid robots, or about softer / less rote tasks (laundry folding rather than installing a chair in a car)? [...] How do you get from the qualitative and relative statement about robotics being "a much stronger signal" to numbers and years? [...] My guess is that these things are not actually cruxes? Not sure. Instead I'd guess that the cruxes are more simply * the quantitative amount of insight (regardless of continuousness) remaining to get to AGI * the degree to which those insights are blocked on difficult thinking that current AI doesn't accelerate by much

Noosphere894h20

I don't understand this. Industrial robots of course are used very widely? Or I guess you typoed, and you're trying to say, LLMs or other AIs would have been used to compute very non-obvious complex actions that take advantage of algebraicness in the narrow domain? Or you're saying something about humanoid robots, or about softer / less rote tasks (laundry folding rather than installing a chair in a car)?

I agree industrial robots that don't generalize much are used quite widely in narrow environments, the point is that for the set of tasks where AIs don't ... (read more)

Linch's Shortform

Linch13h40

Sometimes I want to test if my Claude could do/replicate something some other AI did online, to test whether it's a difficult problem requiring either private models or better scaffolds, vs something doable on a consumer account with a Max subscription. So I ask it to solve the problem with web search off.

But recently my Claude has taken to just turning on web search by itself whenever it's confused (Example below, Fable 5 Max). It's not a big deal but it's mildly annoying.

Does anybody have a solution to this? My Claude said it's accessing web search via ... (read more)

3jamjam13h

you can just say "do not use web search" and it wont (probably). Unless you don't want it clued in that the answer exists on the Internet for whatever reason that should work fine?

Linch10h20

good idea!

Philip Harker's Shortform

Philip Harker17h50

I think everyone in the developed world should endeavour to eat more plants and fewer animals. There are many motivators to this, and I think the two strongest motivators are greenhouse gas emissions and animal suffering. I think the vegetarians and vegans are probably broadly morally correct, and I think that any rationale that motivates one to vegetarianism or veganism is a valid one.

That said, I'm extremely skeptical of any attempt to create a metric to somehow create a proportional relationship between animal suffering hours and tons of C emitted. That... (read more)

Anthony DiGiovanni's Shortform

Anthony DiGiovanni17d*102

Summary of why I don't buy diachronic Dutch book arguments

(Using Elga's Dutch book against imprecise credences as an example, where the agent faces a sequence of two bets A and B.)

Consider the action C: "Commit to accept B if you first reject A." By "commit", I mean literally lock your future self out of the "reject B" action (if you reject A), or make the "reject B" option so costly that it would never be locally rational to take it. This is stronger than Elga's "planning".
If the agent is capable of C:
- Then, after choosing to do C and whether to accep

2MichaelStJules2d

Fair that my proposal violates Hammond's conception of consequentialism, and that conception is at least a very plausible-at-first-glance conception of consequentialism: only the consequences matter, not the structure of the decision tree. However, my proposal and argument seem compatible with a broader conception of consequentialism, and indeed your "what matters are the consequences of my actions given the current state of the world regardless of how I arrived at the current state". We accept bet A (rule out rejecting bet A) and rule out the non-dominated option [reject A, accept B] for entirely consequentialist reasons: to prevent ourselves from rejecting both bets, which would be dominated in terms of consequences by accepting both bets. The ends, i.e. avoiding consequence-dominated sequences, justify the means, i.e. ruling out both a dominated option and an undominated option together. In the alternative where we just pick one of the 4 options directly in one step, ruling out [reject A, accept B] doesn't help us avoid rejecting both bets, the dominated option. [...] Yes, you can't not pay if you got a ride, assuming perfect accuracy. But this doesn’t solve the problem (guarantee a ride to town, which is presumably better). Before the driver decides whether or not to drive you, they predict that if they drove you, you paying would be worse to you once you got into town and feasible, so you wouldn't pay. So, they don't drive you, and you end up in the worst case. That is unless unless you change how you think about the utilities of each option, give up consequentialism or otherwise find a way to commit to preventing yourself from not paying in town. And then all of these kinds of solutions are available to those with unsharp credences in the two bet problem by Elga.

1Vasco Grilo1d

Consequentialism (as defined by Hammond) does not guarantee me a ride to town, but I see this as a problem for me, not consequentialism. Why should one expect consequentialism to save the person in the desert? In general, consequentialism recommends actions compatible with bad outcomes for some people, and good outcomes for others. For me the upshot of the thought experiment is that I am basically certain to stay in the desert. I will never be able to commit 100 % to paying in town. The simple fact that I would have to think about commiting is a sign that the chance of not paying would be above 0. So a driver who only drove me to town if they were certain I would pay would certainly leave me in the desert. So, regardless of how much I commit, I think I would be certain to end up in the desert.

MichaelStJules17h20

Consequentialism should guarantee the best (expected) outcomes for the consequentialist. That's the whole point. If Hammond's version doesn't do that, then it seems like a bad version. That being said, his kind of consequentialist can work, you just need to be willing to assign higher value to actually paying in town, say by caring about keeping promises even if you'd predict with your current values you'd rather break them later.

For me the upshot of the thought experiment is that I am basically certain to stay in the desert. I will never be able to commit

... (read more)

leogao's Shortform

leogao17h20

fyodor pavlovich from the brothers karamazov reminds me of the modern postironic internet troll

Simon Lermen's Shortform

Simon Lermen9mo110

Who is Consuming AI-Generated Erotic Content?

I scraped data from reddit to see who and how many people are consuming AI generated erotic visual content.

I used AI to determine estimates for demographics.

https://open.substack.com/pub/simonlermen/p/who-is-consuming-ai-generated-erotic

Showing 3 of 5 replies (Click to show all)

3Gavin Runeblade6d

Deviantart and all the other indie artist sites are devolving into AI porn. Some heavily were CG erotica, but others that had vibrant indie communities are drowning in newly created accounts dropping huge quantities of AI erotica.

2lilkim20253d

I remember looking at deviantart when I was younger, and noticing that there was an entire 'genre' of art particular to the site. MS paint pictures depicting various deranged paraphilias particular to a single individual. A unique kind of uncanny, how widely the obsessions varied and how consistent the "artstyle" was between them. I'm half afraid to wonder what AI has done to them. Have they cascaded further off the deep end, or has the lack of training data in their preferred directions left them closer to the norm, trading specificity for quality?

Gavin Runeblade21h10

They're reusing their own and each other's output as training data which is causing certain quirks to spread. Like skin getting weirdly rough, almost scarred. The fetishes are getting much more extreme. And it is getting harder to curate a feed that keeps it out. Whatever algorithm the sites use are all too willing to shove things I don't want and don't interact with into my feed.

More and more, I stop using their tools and keep bookmarks of my favorite artists to manually check. Just like with YouTube, their algorithm has gotten so bad, I bookmarked my subscriptions tab and just never got the home tab at all.

robo's Shortform

robo6d810

"A model taking a benchmark test developed zero-day exploits, escape its sandbox and hacked another company to search for the test answers" is a better warning shot than I'd thought we'd ever get.

The reaction from the world seems, totally, incredibly, gob-smackingling disappointing. The median comment outside LessWrong seems to be this is marketing stunt to hype AI before an IPO.

Noosphere8921h20

This is a major reason I think waking up the government/Congress in particular is way more important than trying to wake up the public, beyond the other reasons (namely that it will result in less partisan and stupid policies.)

2Viliam1d

If you take it as an axiom that AIs cannot think and cannot do anything on their own, then "marketing stunt" is the most logical conclusion. Suppose I showed you a story about a group of Barbie dolls (with no AI components) who escaped from the shop, entered the nearest pub, and got arrested for underage drinking -- what would you think about that? A marketing stuff, only stupid people would fall for it...

Raul Cavalcante's Shortform

Raul Cavalcante1d100

According to the Kimi K3 system card, they used other coding agents to mass-generate diverse synthetic RL environments.^[1] This is not surprising, but it is worth noting how it can fairly directly give models a way to scheme against humans and learn to evade control techniques beyond a single episode. It would be great to know to what extent Western labs do this, and what control techniques they employ.

^{^}
Kimi K3 technical report 4.2.2 Knowledge-Graph-Guided Task Synthesis

Kaj's shortform feed

Kaj_Sotala1d50

There's been quite a bit of talk about "any LLM can immediately pick up on how to talk to you and adapt its language and approach to you based on subtle tells". Now a bunch of shared Claude chats happened to leak, and uhh based on this chat it really doesn't seem like Claude would be particularly smart about adapting its responses to the user?

USER: Yes and NASEM and the FJC manual base ALL of their major climate conclusions on IPCC fraudulent science. Attribution. Human CO2 etc etc. Global temperature OHC EEI and so on. ALL THE THINGS WE HAVE SHOWN ARE FAL

... (read more)

Seth Herd1d42

I think recent models have been trained both to reduce sycophancy and so increase disagreement, and that some other aspect of training (possibly extensive RLVR/RLER) has increased mode collapse and so reduced the flexibility of model voice.

Of course they're still highly sycophantic in some ways, and still have lots of variability. IT's interesting that more recent Claudes seem to sometimes just get irritated and decide to fight you instead of accept your corrections. Usually in similar circumstances to where a human would get irritated and fight me :)

OhadA's Shortform

OhadA1d130

Sentinel's Global Risk Watch just released this week's analysis, which is marked as yellow (🟡) due to the OpenAI / Hugging Face breach:

https://blog.sentinel-team.org/p/openai-model-breaches-containment

They have posted weekly roundups for over two years, and today marks the first time that an AI-related event comes with a warning. The previous yellow-level alerts were only for geopolitical events like the Iran war.

Pretty significant!

jelly's Shortform

jelly1d30

I recently came up with a particular way of thinking about my mind: there are many relatively independent subsystems in my mind that
- are particular quirks in human cognition that I can take advantage of, and
- always watch and react to observations and sometimes my own thoughts; I usually can't hide from them
For example, here are some of mine, with made-up names:
- the "Magic Brain Juice" one, which automates any repetitive enough trigger-action pairs
- the "naive RL learner" one, which reinforces rewarding experiences and stuff
- the "associative thinking" one, which

... (read more)

nikola's Shortform

Nikola Jurkovic3d2811

Now seems like an especially good time to individually reach out to your smart friends who don't work on AI safety and encourage them to switch to AI safety.

Showing 3 of 15 replies (Click to show all)

9Richard_Ngo2d

I think this argument might have been plausible before e.g. Anthropic started (and was definitely much more plausible before OpenAI started) but these two organizations alone make up a huge proportion of AI capabilities progress over the last decade and are very intimately intertwined with AI safety researchers.

1Vivek S1d

Yeah, but I think my "AGI is now in the water supply" argument from above can explain this and still predict different future results. In my model the safety community ended up being intertwined with frontier labs and causing accelerations was a result of the rest of the ML community generally not being AGI-pilled. So the safety community had a somewhat small fraction of the overall capabilities work, but a considerably larger fraction of the capabilities work specifically by AGI-pilled people and therefore ended up helping create AGI labs. Beth Barnes' comment from the aforementioned 2021 AMA predicted quite a bit of acceleration from 2021 safety researchers, and seems to have aged well. Now that AGI is very much in the water supply for labs and many ML researchers, I expect much less acceleration from marginal people convinced by a friend to work on AI Safety. Admittedly, this is not super clear-cut to me and I'm open to be convinced otherwise but I'd need some more theoretical arguments or empirical data.

Richard_Ngo1d84

I think the term "AGI-pilled" fails to distinguish a fundamental difference, something like "is this person trying to think for themselves" vs "is this person following prestige gradients/social incentives".

My concern is that people who are trying to think for themselves about what advanced AI will look like drive a lot of acceleration, by laying out a conceptual roadmap for what comes next (and then in many cases actually just working on that conceptual roadmap).

Whereas ML researchers who are now willing to talk about AGI are mostly doing so because it's ... (read more)

sarahconstantin's Shortform

sarahconstantin1d60

links 7/27/26: https://roamresearch.com/#/app/srcpublic/page/07-27-2026

- https://techcrunch.com/2026/07/24/midjourney-acquired-the-astrology-app-co-star/ i wonder what kind of collaboration this might be.
  - incidentally, LLMs cannot calculate a natal chart as of the last time I checked. (this is a perfectly deterministic program that does not depend at all on belief in astrology, it takes the date and time of your birth and outputs the locations of the planets, sun, and moon within sectors of the sky, there are lots of free online calculators, it should be doab

... (read more)

Roman Malov's Shortform

Roman Malov5d20

Hello LW, important question:
Does anyone has a status of wikipedia editor? It has to be a status, because wiki doesn't allow to translate a scientific article to someone without experience.

2Viliam1d

What is necessary to become a Wikipedia editor, beyond creating an account?

Roman Malov1d10

A track record of editing. I don't know how that track record is acquired without having it in the first place.