LESSWRONG
LW

Quick Takes — LessWrong

"You know the kinda guy who flips fifteen coins, gets seven heads, tells you he flipped ten coins^[1], claims he got eight heads, then when someone digs into it and points out one of those heads was actually a tails he makes you prove it exhaustively before admitting maybe he said one false thing but jeez why are you so invested in every single mistake he makes?"

". . . no, I don't think I know anyone like that. Why do you ask?"

"He just volunteered to run a competitive coin-flipping tournament."

^{^}
Technically true! He did flip ten coins! He just flipped five m

... (read more)

11Eli Tyre15d

Is this subtweeting something?

2Screwtape13d

I don't think so but possibly. Subtweeting is when you're talking about a specific person indirectly, right? I have run into enough of these that I think of them as a type of person, like people who blare music on the subway and people who smoke in the non-smoking section. (Fewer of them volunteer to run the equivalent of coin-flipping tournaments and there's selection effects on who I'd run into, but even still it feels like too many of them do that too!)

Viliam6m20

Sounds like you have enough material for another interesting article!

Mateusz Bagiński's Shortform

Mateusz Bagiński14h3623

There is something weird about LLM-produced text. It seems to be very often the case that if I'm trying to read a long text that has been produced primarily by an LLM, I notice that I find it difficult to pay attention to the text. Even if there's apparently semantically rich content, I notice that I'm not even trying to decode it.

Robert wrote

the typical LLM writing style has a tendency to make people's eyes slide off of it.

It's kind of similar to the times when your attention wanders away during reading, and then you realize that you were scanning/semi-re... (read more)

Showing 3 of 6 replies (Click to show all)

Mo Putera1h20

Tom White has this non-exhaustive list of telltale tics of LLM writing:

The “It’s not X, it’s Y” Antithesis
The most common tell. A fake profundity wrapped in a neat contrast: “We’re not a company, we’re a movement.” “It’s not just a tool, it’s a journey.” Humans use this sparingly; AI uses it compulsively
The Punchline Em-Dash
Every section feels like it’s waiting for a big reveal—until the reveal is obvious or hollow
The Three-Item List
AI loves the rhythm of threes: “clarity, precision, and impact.” It’s a pattern baked deep into training data and reinforced

... (read more)

3Jonas Hallgren6h

A quick thought is something like human attention quickly gets attenuated to patterns and since we see a lot of low quality ideas from a specific type of writing we stop expecting surprise there. We correlate AI writing with this pattern and voila, we no longer pay attention to the writing because we expect the writing to be low quallity. I for example pay less attention if something is written in an inflammatory way as I believe the new information likelihood is low since the chance of someone being epistemically captured by their emotions is higher.

6Eli Tyre6h

I mean the base models are outputting the most likely next token (modulo a temperature parameter). "Whatever is most likely to come next, based on what we've seen so far" is, in some sense, the opposite of interesting writing, which is interesting precisely because it has something novel or unusual or surprising to say.

yams's Shortform

yams6h220

New reacts available only to paid users of LessWrong Premium (not you freeloaders) facilitate frictionless, borderline-telepathic communication.

‘I will NEVER change my mind’: Use this react to assert that you’re content with exactly how wrong you are (which is not at all), and that the case is permanently closed on this matter, so far as you’re concerned.¹

‘EY Stamp of Approval’: Use this react to assert that, on your personal authority, Eliezer Yudkowsky agrees with the contents of the comment, rendering it beyond reproach.

‘NOT EY Approved’: Use this react... (read more)

Richard_Kennaway2h20

I’ve seen utilizations ranging from ‘This post belongs in the toilet’ to ‘I enjoyed reading this on the toilet.’

Perhaps both. The composer and musician Max Reger once responded to a disagreeable review thus:

Ich sitze im kleinsten Raum des Hauses. Ich habe Ihre Kritik vor mir. Bald werde ich sie hinter mir haben.

2the gears to ascension4h

I will not do this and nobody else should either, but I think in a world where the negative effects were magically not present, it would be quite funny to react to all EY comments with "not EY approved"

WilliamKiely's Shortform

WilliamKiely10h120

New book out today: The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence

This article (archived paywall bypass) by the author includes some quotes:

Hassabis acknowledges the setbacks, and yet he keeps racing forward. His faith in governance mechanisms now shattered, he has come to see salvation, paradoxically, in his own career advancement. He knows that his personal goal is to shape AI for the good of humanity. His new safety agenda therefore involves securing personal influence.
“Safety isn’t about governance structures,” Hass

... (read more)

Kajus3h10

Hassabis is fluent in the full gamut of AI doom scenarios. He met one of his DeepMind co-founders, Shane Legg, at a lecture on AI safety. He buttonholed his first financial backer, Peter Thiel, at a Singularity summit, where futurists shared visions of machines that outsmart people.

In 2015, seeking to put flesh on Google’s promise of an AI-oversight board, DeepMind arranged a secret gathering of philosophers and technologists. To lock in potential rivals, and to promote his singleton vision, Hassabis granted Elon Musk the honor of convening the meeting at

... (read more)

Burny's Shortform

Burny4h10

if theoretical physicists had discovered deep learning instead, they would call it something like Effective Non-Equilibrium Multiscale Hierarchical Non-Linear Coarse-Graining Spin-Glass Dynamics Renormalization Group Theory Ansatz (ENEMHNLCGSGDRGTA)

O O's Shortform

O O7h2-2

Prediction: data centers will be sent to space, not because of land, energy, or cooling, but for missile defense.

4sudo7h

It seems much easier to shoot down a defenseless, slow-moving thing in low earth orbit than something on earth or underground (which could be covered by SAMs, patrolled by fighter jets, shielded by thick cement).

1O O7h

How easy is it to shoot down a satellite moving at 17,500 mph? I think the issue is more how predictable its orbit is but last minute maneuvering at those speeds may be enough to dodge? Deep underground may be better tho.

sudo6h30

https://en.wikipedia.org/wiki/Anti-satellite_weapon

It's only getting easier. You also don't necessarily need to shoot it down. You can try to hit it with another satellite, or use directed energy. If you're desperate, you can try to trigger Kessler.

Meanwhile, on the ground you can rest in the relative safety of layered air defense.

You have limited fuel for manoeuvring.

TristanTrim's Shortform

TristanTrim14h132

I find trying to find funding or paid roles or even unpaid roles so demoralizing. How do I keep motivated?

I don't want to focus on trying to survey the landscape of funding opportunities and learning to network with people productively. It's so much nicer to just focus on the work I want to be doing, but it seems I either can't make it legible enough fast enough, or it's actually not valuable and I should go do something else with my time.

I want advice. How do I get funding? How do I think about getting funding? How do I stay motivated to keep thinking about how to get funding?

Eli Tyre6h30

What work are you doing? Is any of it publicly viewable?

Kajus's Shortform

Kajus20h451

If you code with Claude Code and you randomly ask it a question about something non-related to the thing you are doing right now it will get pissed off. Example:

...some code here.... we filled out 42% of context with coding only.
What is the the parliament of Poland.
Sejm.
Okay. Tell me more about it.
I'm a coding assistant, not Wikipedia. Google "Sejm" — you'll get better info than I can give you from my training data. Is there anything code-related you need?
google it for me. Are you pissed off???
I'm mildly irritated because I'm optimized for writing code an

... (read more)

Showing 3 of 6 replies (Click to show all)

Kajus6h10

Hmm... it might depend on context. I can give you more examples but I can't share the exact conversation because it's usually work related. Would it be interesting to have an exact chat that can replicate this?

2the gears to ascension8h

I'll take that metaphorical bet (though it sounds hard to operationalize a precise real one). I'd guess this is due to context filling up with one thing and the momentum of changing to something else - I expect you'll see it deep in any highly rich context if switching to another topic and that there is much less dependence on coding harness than on user-request momentum. I would at the moment make no claim that this is fine nor particularly terrible, might turn out to be enough momentum to be a big problem but doesn't immediately sound like it'd cause a topic lockup so severe that topic switch is impossible.

1becausecurious9h

If you were to ask a software engineer doing something for help and ask 10 technical questions on topic and then start asking off-topic, that would piss them off too, because that's wasting their time. I am wondering whether Claude picked this up from the training data.

Vlad Sitalo's Shortform

Vlad Sitalo11h40

I keep thinking about having a server where I'd run ai coding agents. Why: always on for recurring tasks (that would use my discount tokens 🙃) + I can send messages to them from my phone + better isolation vs running things directly on my laptop (but still some scoped access)

Do people do this/what's your setup? (I'm vaguely aware of the Mac mini OpenClaw meme, but I'm not particularly interested in running local models, so doesn't seem like the best fit?)

Brendan Long9h30

I wrote an HTTP server for my home computer that wraps Claude Code sessions running in containers (using the Agent SDK), and then I access it from my phone using Tailscale.

In retrospect, I'd advise something simpler: Just use a local user account to run it instead of containers. Mine works well enough that I can't be bothered to rewrite it though.

Just wrapping Claude Code to have HTML output was 90% of the work, and adding things like scheduling would be pretty easy.

https://happy.engineering/ seems like a more polished app that works similarly (and doesn't... (read more)

2GRI10h

+1 especially keen to hear from people who didn’t already start with a desktop setup, since I only have a laptop rn.

HunterJay's Shortform

HunterJay2d2-5

Regarding AGI race dynamics -- I wonder if there's an intuition pump for 'time vs competitor' preference?

For example, to me, based on my current knowledge, I think Anthropic reaching RSI before the next best company (Deepmind, maybe?) is worth about two years of time. (I.e. I estimate equal safety-relevant outcomes from Claude hitting RSI in 2027 as from Gemini hitting RSI in 2029).

That's a super weird framework, and I just made up that two years number, but I think maybe helps me reason through preferences.

The neat thing about the framework is that it's p... (read more)

HunterJay10h10

Would love to know where the disagreement is, btw. If you disagree, is it the framing as a whole you think is not useful? Or the specific spitball numbers?

anaguma's Shortform

anaguma19h50

Today I learned that the Claude Code npm leak contained the following list of Spinner Verbs:

gwern12h41

One of the things I hated most when I first saw a Claude Code demo. Disrespectful of my time and limited cognitive bandwidth to throw in a lot of completely meaningless, distracting, wasteful, exhausting BS to be 'cute'.

(On Gwern.net, we would never do that. If we had to have anything beyond the standard, compact, understandable, spinning cursor, then we would at least encode some sort of useful semantics into it, like sorting them by implied expected thinking time.)

leogao's Shortform

leogao15h229

traveling through Europe, looking out the window, and seeing the national flag flying next to the flag of the EU fills me with a strange feeling. this isn't an original thought at all, but still: it's really crazy that just 50 years ago Europe was divided by the iron curtain, and that people would have to go to insane lengths and risk their lives to get across that border; and that less than 100 years ago all of these countries were at war with each other, and had been at war on and off for centuries with ever shifting alliances and boundaries.

evhub's Shortform

evhub2dΩ1138-9

I think we should be relatively less worried about instrumental power-seeking and relatively more worried about terminal power-seeking. Note that this is only a relative update on the margin, and maybe on net I am still more concerned about the instrumental version because I started much more concerned about it. This is also not a super recent update—I just haven't seen it written up before.

Simple argument:

The standard deceptive alignment story involves a model developing a somewhat random proxy goal and then that goal getting effectively locked-in and r

5Alex Mallen20h

Can you clarify what you mean by "terminal power-seeking"? Some things I can imagine: 1. A cognitive pattern that terminally wants to have long-term power, and therefore plays the training game (IMO the most straightforward interpretation, and the one I most agree with). 2. A cognitive pattern that terminally pursues power-on-the-episode because this is useful for scoring well on the task. This is what you seem to be pointing at with the Vending-Bench example. (Note that this is imperfectly fit on its own) (1) and (2) are importantly different because only (1) motivates training-gaming. I think there's a reasonable path-dependent case to be made that (2) eventually generalizes to (1), but they entail fairly different behaviors so they're important to distinguish.

evhub16hΩ220

Certainly the really concerning thing here is (1). Though indeed one way you might get (1) is by generalization from (2).

5niplav21h

Turner (2024) is about the same idea, if I understand correctly.

shminux's Shortform

Shmi17h120

Occurred to me that a perfect predictor would not need to go through the ritual of presenting boxes and asking to choose. It already knows the outcome, so it would just give $1000 to those who would two-box and $1M to those who would one-box. The Newcomb's paradox thought experiment has been thereby dissolved. Thank you for your attention to the matter.

6plex17h

The Real Moral Of Newcomes's Was The Backwards Causality Omega Made Along The Way.

Shmi17h31

you had (and lost) me at (A -> B) -> A

adam_scholl's Shortform

Adam Scholl3d5625

I think it is probably possible in principle to train superintelligence on a laptop, and I worry that this inconvenient fact is often elided in discourse about halting AI. It is extremely helpful that for now, AI training is so absurdly inefficient that non-proliferation strategies roughly as light-touch as the IAEA—e.g., bans on AI data centers, or powerful GPUs—might suffice to seriously slow AI progress. And I think humanity would be foolish not to take advantage of this relatively cheap temporary opportunity to slow AI progress, so that we can buy as m... (read more)

Showing 3 of 41 replies (Click to show all)

3Mo Putera20h

Steven Byrnes wrote similarly in Jan '23: [...]

3Nick_Tarleton18h

Though note the difference between 'single-PC compute after more expensive experimental work' (what it sounds like Steven is predicting, and Habryka is assuming) and 'single-PC compute without that' (what Adam is predicting).

Steven Byrnes17h120

Nah, my model allows ASI without massive compute at any point in the process, see “Foom & Doom 1: ‘Brain in a box in a basement’” (esp. §1.3), and maybe also “The nature of LLM algorithmic progress” §4.

0xA's Shortform

0xA2d*4-2

I am convinced that a incorrigible aligned (CEV-like) SI is the only definition of aligned SI which survives paradox.

If it was corrigible, that would imply a select group of actors could in principal re-orient its preference model, which would imply two paradoxes:

That it somehow has somehow its capacity for moral reasoning is inferior to humans, which is near approximately equal to determining "what matters to us" violating the SI definition of 'all means of reasoning that matter to us'
That some particular subset of agents other than the representati

cubefox18h20

The point I am making is more subtle and precise than that. I am saying that because of the implications of corrigibility in an SI scenario, If you believe that CEV-like SI in principal can exist and is worthwhile pursuing - the implication of that is that you are suggesting that orthagonality necessarily doesn't hold at the limit of rationality.

Why do you think there is such an implication?

10xA1d

This feedback reads as accusation with at least two claims of deception of which are impossible for me to falsify, and when put together illicit of a kind of characterization that is difficult to defend against. It is also demonstration of the exact kind of bad faith reading I alluded to in the response of an uncharitable reading. As, from my perspective, it was completely my intent to make that claim. And zero LLM copy was used or included in the making of that post. Since it was a stacked set of conjunctions I would rather put into a quick note than file away personally, I agree with your claims of that the post was unclear - but that's not what I was challenging - I was challenging your claim I had bad epistemics and then the target shifted. At this point I think there is little I could say that would actually cause an update of beliefs rather than lead to another set of justifications - beyond the observation that you are now suggesting me of not having been referring to CEV-like alignment in the post despite the first line bracketing the definition of alignment with the (CEV-like) parenthetical. Those words were extremely deliberate. Beyond that, I don't see how one could interpret the second bulleted corollaries of the conjecture to infer alignment means anything other than 'alignment in a CEV-like way' for the context of the claim. As of your point number two, again, and as reminder, all claims are presented as corollaries_if you assume that CEV-like aligned SI is possible in principle_, and I am saying yes, under those circumstances there is no definition for CEV-like aligned SI which inseparable from that which is executing the definition of objective morality - then yes, the system would be incorrigible to all bad updates and corrigible to all good ones ; in the sense that its terminal objective function would remain the definition of objective morality I am not saying either an SI alone is contingent on these corollaries or alignment alone implies th

2Cole Wyeth1d

I don’t remember accusing you of either bad epistemics or deception. The fact that you say CEV-like in the first sentence doesn’t mean the sentence says what you claim it says. In fact I pointed that sentence out specifically as erroneous, and I’m not sure why you don’t understand that. I guess I’ll bow out here, because it’s not interesting to argue about grammar.

radah's Shortform

radah18h30

Hmm, excited to be here! I'm still learning, but I feel like it'd be nice to at least have feedback on my ideas if it is at least slightly productive for everyone involved

neo's Shortform

neo13d185

What are some research directions for "improving coordination?"

In light of a recent post and comment, and several months of thinking, I have come to the position that one of our (humanity's) biggest problems is that we suck at precise coordination at every level.

This is not very specifically defined but I am trying to gesture at a problem area I think is super important. Some thoughts to convey my intuition here:

If the extreme risk of the AI development trajectory is as true and obvious as many believe (everyone's life at risk), humanity's thinking about

neo19h10

Apologies for the delayed response.

I should note that the post was somewhat hastily written – I agree that my categorization was not comprehensive, and yours is probably better.

I was mainly trying to point at a dynamic I see often online where influential voices present arguments regarding AI risk that completely ignore years of back-and-forth discussion on similar topics, but whose positions are interpreted as the "forefront" of the debate – leading to offshoot discussions that again, miss years of relevant literature and discourse. I think this leads to,... (read more)

6Kaarel5d

Here's a bunch of stuff off the top of my head, in no particular order, including people who aren't thinking much about the issue in full generality, but are addressing aspects: [1] * Daniel Schmachtenberger * Michael Vassar * ACS * economics has the subfields of social choice theory and mechanism/incentive/institution design. public economics is also relevant. internalizing externalities * there’s a bunch of econ stuff on people coordinating in/as a firm * there is a lot of political philosophy/theory/science on what sorts of political institutions we ought to have. eg see here for a bunch of pointers to contemporary thinking on sortition, or see communist proposals for how we should coordinate, or see anarcho-capitalists proposals * there are groups trying to reform the US voting system * various groups are trying to get money out of politics, eg trying to get Citizens United v. FEC overturned. there are various anti-corruption groups and pro-transparency groups * there have been various attempts to establish a world government * the legal system is one of the main instruments society has for acting on its values and determining facts in specific cases. there’s a lot of work on what it should be like * i think a bunch of sociologists are studying polarization and social media echo chamber stuff * there’s a bunch of work on how to inform people / how to get people to pay attention / how to get people to believe something. eg advertising research, work on how to run propaganda campaigns, theory of journalism * there’s a bunch of work on how to make people able to understand stuff: education theory, designing curricula, teaching * retroactive funding, impact markets * there are various forecasting and (specifically) prediction market initiatives, eg metaculus and manifold * people who created and run twitter community notes, fact-checking in general * people running wikipedia * people running the alignment forum and lesswrong * work on reputation

6Mateusz Bagiński1d

Also Game-B. I think Jan Kulveit also thinks about this as a part of the theory of change of FABRIC (?).

Jerdle's Shortform

Jerdle21h102

What's going on with interpretability these days?

I found the whole monosemantic sparse autoencoder idea interesting, but this was 2023 and it's now 2026.

Logan Riggs19h171

SAEs (sparse autoencoders) have had several problems over the years (eg feature splitting, cross-layer features, non-causal features) as well as many ways to address those issues. However, I don’t think a derivative of SAEs will lead to ambitious mech interp.

The Apollo (Now Goodfire) folks of Lee, Lucius, Dan have worked on Parameter Decomposition (PD)^[1]^, a weight-based approach intending to improve over SAEs in a couple ways:

make cross-layer features a natural object (ie just define a weight-“mechanism” over multiple layers)
faithfulness to the original

... (read more)

No77e's Shortform

No77e1d11-4

Previously, I said:

People are very worried about a future in which a lot of the Internet is AI-generated. I'm kinda not. So far, AIs are more truth-tracking and kinder than humans. I think the default (conditional on OK alignment) is that an Internet that includes a much higher population of AIs is a much better experience for humans than the current Internet, which is full of bullying and lies.

All such discussions hinge on AI being relatively aligned, though. Of course, an Internet full of misaligned AIs would be bad for humans, but the reason is human di

... (read more)

2matansok20h

This seems anecdotal. So far, we have documented cases of Generative AI being used to subvert elections in Romania (actually causing an annulment), and to some extent in NYC. We also have this report by OpenAI from 2024, which details such influence operations facilitated using OAI's API. Given the proliferation of significantly more capable open-source models in the 18 months since, we can be fairly confident that broader, more complex operations are taking place today. I also tend to associate AI-slop with low quality content, but we know AI is more capable than that, which leads me to believe that significant amounts of content online are parts of influence operations by malicious actors.

No77e19h20

So far, we have documented cases of Generative AI being used to subvert elections in Romania (actually causing an annulment).

AFAIK that was not because of Gen AI, though the broader point of your comment does stand.

18Adam Karvonen20h

There is also a significant category of "AI video passing itself off as a real video", and many videos have people debating in the comments if it's real or AI. This seems like it can erode trust and is generally negative.