Quick Takes — LessWrong

LESSWRONG
LW

There is something weird about LLM-produced text. It seems to be very often the case that if I'm trying to read a long text that has been produced primarily by an LLM, I notice that I find it difficult to pay attention to the text. Even if there's apparently semantically rich content, I notice that I'm not even trying to decode it.

Robert wrote

the typical LLM writing style has a tendency to make people's eyes slide off of it.

It's kind of similar to the times when your attention wanders away during reading, and then you realize that you were scanning/semi-re... (read more)

Showing 3 of 18 replies (Click to show all)

Mateusz Bagiński1h20

Maybe. I would still expect them to either feel like something's off (perhaps being unable to verbalize exactly what's off) or at least that their attention is sliding over the wall of text in a similar way (at least after some time of interacting with LLM text).

0icely4h

I find this unsurprising and it's odd to even pose as a question. Reading older confident GPT-3-quality LLM's that don't know what it's talking about, reading tons of mediocre spam by online people copypasting LLMs, plus of course the default style being close to journalist quality sludge that was already mediocre when humans were writing it, don't give a good impression of the quality. If you actually want to read it request a denser or different style.

3XelaP11h

Nitpick: I like 3 item lists. Almost everyone is regularly not specific enough. It's your brain's superpower, concrete examples are possibly how humans understand most concepts, and you're probably still underestimating how specific you should be after reading all that. [...] Why three? Well, in practice that seems to be enough that you have a handful and aren't overindexing too hard, got some variety, and is at the edge of difficulty to come up with on the fly. LLM bulleted lists do feel less eye-glazey; I think they're denser but maybe it's just more skimmable. The following list was not intentionally made to have 3 examples - that's just I happened to do in a medium length of time. Doing the first example was easy, the second I thought of by the time I finished fleshing the first out, and the third required some search. Ideally you push yourself to the point where you have to sit and stare at nothing trying to think at least once. examples of examples being good * Very often when doing a math problem I go "Oh, I don't need to write down a small special case to work with, that won't teach me anything!" and I'm wrong most of the time. (less often nowadays but only because I get the premonition of preventable wasted time) Sometimes people (especially if anxious or depressed) think things like "Everyone hates me". If you ask yourself "name three times you've observed someone acting like they hate you" then often (if you get any at all) the examples are just like... obviously really weak, like "they seemed mildly bored once when talking to me" or "I spilled a drink on them and they didn't make a big deal but they looked a little sadder after it". * A trick Feynmann said he used: when mathematicians would tell him about math was imagine an example of an object satisfying the premises and then check the conclusion against it. To me this is like... if you don't do this then like how are you even thinking about math well enough and for long enough to be teachin

yams's Shortform

yams6h260

Meanings of political identities shift dramatically based on context, and you can't manually confirm the beliefs of everyone present at your 'gathering of people with x political identity'. To the extent that your political identity is based on Real Beliefs with Real Consequences, you should expect not to have much in common with many other people who declare the same identity when you move to a new place (or corner of the internet).

Example: In rural Southeast Texas, Confederate flags are a common sight, and my geometry teacher once told us about a cross b... (read more)

Showing 3 of 4 replies (Click to show all)

2yams1h

I don't understand this. Can you say more?

2Garrett Baker1h

I submit that the problem you are encountering is operating too much on simulacra level 3. I honestly don't give a rats ass what your "political identity" is. I care about your beliefs about the effects of different policies, your current beliefs about the world, and most of all your reasons for believing those things. Insofar as your "reasons" are in fact you self-modeling as "leftist", "centrist", "libertarian", or some other label like that, you are clearly operating on simulacra level 3, and I stop caring. That is to say that insofar as the majority of people have one or more of the items on that list of absurd beliefs you gave because they genuinely believe such things, they are dumb, and the only reason I'd care what they have to say is some morbid curiosity, or of course to argue against them (or both). Insofar as the majority of people have one or more of the items on that list of absurd beliefs you gave because we simply lack a shared vocabulary for political identities, I again don't care what they have to say. If they are actually trying to communicate to me that they think of themselves as a centrist, well, that claim has no impact on reality whatsoever, so who cares? The same goes for people with stupid claims which read leftist. The thing is that we do have shared vocabulary for things in the real world (excepting many technical terms, but one can just use Wikipedia or Claude for those), and there are not active social wars over what those real world words should mean (that laypeople ought to care about). If you stick to simply precisely stating your beliefs and values (and reasons for having them), not only will you avoid the upsetting result of having others believe you are The Dreadded Outgoup!!! but you will also avoid meaningless conversations about which Hogwarts house political identity you are.

yams1h40

You wrote this comment in an adversarial tone but I Just Agree With You.

Indeed, this is an alternate formulation of the thesis of my post, and even uses language I used when characterizing the post itself to someone in the office ~2 hours ago.

Hastings's Shortform

Hastings2h20

has anyone had success coding while baby wearing? without some trick I dont know, they seem to tolerate phone usage but not touch typing, which is a tricksy nudge toward wasted workdays. considering treadmill + standing desk

4Hastings2h

Proposed alternative: switch to a voice to text first programming pipeline. Unknown effects on verbal development

Bill Jackson1h10

I can't attest to trying it when wearing a baby, but I do find that Wispr Flow + Claude Code (avoids needing to type special characters) lets me code with ~100% voice. And it doesn't have much of a learning curve relative to the old-fashioned ways like https://talonvoice.com/ (my old manager used to swear by this).

2Hastings2h

Proposed alternative: switch to lisp which can be typed with just the phone keyboard and two characters, and period replacing the parens, no phone keyboard mode switching

Warty's Shortform

Warty5h-1-1

standard yudkowskian belief is that superintelligence will "fill the lightcone" with something (paperclips if bad, people if good). seems sus to me, I would expect the law to be that life just fills space with itself asap. e.g. animals don't make paperclips, just make themselves

3Eli Tyre3h

All life forms are optimized, by evolution, for self-replication. It remains to be seen if evolutionary pressures will optimize future AIs for self replication, or only for patient values and aggressive power-seeking.

Warty3h10

Yes, it's mysterious. With a singleton AI, selection doesn't apply. So we could only say like, in the long run the replication-only aliens occupy more space than the side project ones. (Naively, it would seem you could still at least "defend your territory" if you spend 1% on fun.)

Still, when I hear speculations of what unaligned ASI will be doing it doesn't mesh. Replicator feels to me an obvious default, some simplicity intuition as they call it.

Like if we imagine the universe across time, feels clean if the right part of the graph is "dumb superintelligent replicator era"

nikola's Shortform

Nikola Jurkovic5d203

AI safety student groups should probably make it easy for their members to see The AI Doc in the cinema. Maybe announce that you'll all go to the same screening on a wekeend, offer to cover tickets, have a thing right after where you talk about it and maybe get free food or something

Showing 3 of 4 replies (Click to show all)

5Elizabeth4d

why? To inform themselves, or to boost the movie's numbers?

Nikola Jurkovic3h20

To see it themselves, I think it's a good movie that could be motivating (and a fun time to watch) to a lot of young people. I think it's especially good for people who are kind of interested in AI safety but haven't fully decided whether to work on it or not (many members of AI safety student groups fit this description).

1Noah Birnbaum4d

Im thinking to see it themselves.

ChristianKl's Shortform

ChristianKl6h10

Does a hypothetical perfect superforcaster who can't articulate any justifications for their prediction have any knowledge?

Showing 3 of 4 replies (Click to show all)

2ChristianKl4h

I see superforcasting as the ability to give answers about how likely it is that a given event is going to happen and answer with a probability. That's not the same skill as coming up with verbal justification.

Roman Malov3h10

Justification = P("Opponent will agree that S is a good justification for prediction" | "I say S as my justification for a prediction"). If there are no division-by-zero errors, that should work.

2Shankar Sivarajan4h

Does the complementary person, whose predictions are shit but can confabulate plausible/convincing justifications for them have any knowledge?

Shortform

lc14d*257

LLMs have protagonist syndrome. They all think they're in a contrived parable about getting reward inside an RL environment built to vaguely resemble the real world. Every situation is part of a story where there's an expected response to the query somewhere out there, even if it's a refusal, or an explanation of why the problem is impossible. Every task treated like an academic exercise as part of a course about economic productivity.

6Stephen Fowler13d

What predictions does this model make?

lc5h20

The priors on what the correct action is are different if you're facing a contrived test vs. a realistic scenario. In an academic setting, if you see a debugging solution that seems like it has a plurality of evidence and just one or two facts that don't make sense, you can be pretty confident that that option is still the result. However, in the real world things are messier, and you would be better resolving your confusion and getting more evidence. This often leaves the AI overconfident and means that it will return early, having identified what would be the solution if it were navigating an RL environment instead of the real world.

9Bronson Schoen14d

To be fair, that’s all they’ve ever seen during mid/post-training! It’s also a really effective way to think during mid/post training.

Max H's Shortform

Max H1d234

There was some discussion recently about the uptick in object-level politics posts and whether this is desirable or not. There's no rule against discussing politics on LW, but there is a weak norm against it, and topical discussions have historically tended to be somewhat meta and circumspect.

I think the current situation is basically fine, and it's normal for amount of politics discussion to ebb and flow naturally as people are interested and issues become particularly salient. That said, here are a couple of potentially overlooked reasons in favor of mor... (read more)

Showing 3 of 15 replies (Click to show all)

Eli Tyre5h20

I think it might be cool if LessWrong had a well developed set of norms for discussing political topics, in particular, these norms were legible, and mods made a point to enforce them.

Politics posts should be tagged as such, and maybe all have a big warning at the top linking to a post outlining our expected norms and standards for discussing politics, and moderation thresholds. This is both a warning to those from other parts of the internet who don't share our epistemic ideals, and a warning to LessWrongers who don't want to wade into this stuff.

4Thomas Kwa7h

Iraq was 32,000 wounded and 4,400 killed, and the US has already suffered hundreds of wounded and 13 deaths in the existing Iran campaign without any ground operations. I'm imagining 100 wounded and maybe another 20 KIA if the US holds Kharg for an extended period, not hundreds of KIA. The issue is it's not really true that the US has air supremacy. Kharg Island is within fiber FPV range of the mainland, and real-time ISR is not required for Iran to track static targets on the island. Plus Iran is still able to launch larger drones and the occasional missile. So holding Kharg really means denying drone launch points on a ~20 mile stretch of the coast, which for FPVs can just be two guys in a bunker. The incentive for Iran is enormous given US's low tolerance for casualties; it's well worth it to launch 20 $1,500 drones to kill one American.

1ACCount8h

Keep in mind that a lot of targets are not "properly fortified", be that infrastructure or military facilities, and suicide drones are much harder to hunt down than ballistic missile TELs. Modern ISR can perform well in a "Scud hunt" scenario, but "Shahed hunt" is a much worse match up.

Oliver Daniels-Koch's Shortform

Oliver Daniels5h10

opinionated ai safety syllabus for people who are "interested in safety" (and maybe even do safety research) but aren't that online / LW-pilled. I think these describe most my "conceptual" research taste (insofar as I have any).

would love feedback / suggestions

Alignment
The behavioral selection model for predicting AI motivations
A simple case for extreme inner misalignment (generally this whole sequence is good, taken with a grain of salt)
Alignment by default
Ngo and Yudkowsky on alignment difficulty + Beliefs and Disagreements about Automating Alignment Re... (read more)

Buck's Shortform

Buck7dΩ20586

Rob Wiblin asked:

What's the best published (or unpublished) case for each of the big 3 companies having the best approach to safety/security/alignment? That is:

Anthropic
OpenAI
GDM

(They're each unique in some way such that someone who cared a lot about their X-factor might favour them.)

My response:

The basic case for Anthropic is that they have the largest number of people who are thoughtful about AI misalignment risk and highly focused on mitigating it, and the company culture is somewhat more AGI-pilled, and more of the staff would support taking actions t

... (read more)

Showing 3 of 8 replies (Click to show all)

RHollerith6h*20

The other reason I trust Deep Mind more than the others is that Gemini lags Claude and OpenAI's services in coding skill, a dangerous capability because over some (unknown) threshold of coding skill, a model will tend to become capable of effective recursive self-improvement.

I could easily change my belief here though especially by my getting more information about Deep Mind.

17Lukas Finnveden7d

Correct me if I'm wrong, but I believe the quote you're referring to is: [...] I think it's inaccurate to summarize this as "publicly committed to a pause if the other leading labs pause".

2RHollerith7d

I accept the correction. I wish there were some other LLM-based service I could recommend to friends, ideally one run by people who would prefer a decades-long pause on AI "progress" and who are only in the market to deprive the other services of revenue and to limit how much the other services can leverage their position as trusted sources of information to influence public opinion on the dangers of AI "progress".

Thomas Kwa's Shortform

Thomas Kwa2d*410

Last year, METR used linear extrapolation on country-level data to infer that AI world takeover would ~never happen. However, reviewers suggested that a sigmoid is more appropriate because most technologies follow S-curves. I just ran this analysis and it's much more concerning, predicting an AI world takeover in early 2027, and alarmingly, a second AI takeover around 2029.

Here are the main differences in the improved analysis:

The original analysis assumed that the equation for AI takeover vs year would be axis-aligned, but this is an arbitrary basis. We n

... (read more)

Yudhister Kumar [Deprecated]6h10

Noting this was posted on April 1, as it won't be immediately apparent to posterity.

robo's Shortform

robo10d*23-2

What would a concrete AI takeover plan look like?

You can smell a chess bot by how quickly they change plans^[1]. Human players act like they have a couple of attack strategies in mind and stick with them. Chess bots change tacks constantly, one move looking like they're moving towards this goal and next move switching to something totally different.

I'm guessing this it how it would be with real-world takeover. Human need a simple grand strategy they can coordinate around ("An amphibious invasion of northern France"). AIs, even very weak ones^[2], have fa... (read more)

Showing 3 of 4 replies (Click to show all)

3robo6d

1a3orn made what I thought was a good reply in a DM. My digestion/riff would be "You're implicitly assuming that planning will be especially cheap for an LLM agent compared to humans. But even medium term planning / resilience to plans going awry is a particular weakness for LLM agents. The first, weakest agents that could pose a risk to humanity might be differentially bad at planning, find it differentially expensive to make good plans, do differentially less planning."

Viliam9h20

Yeah, when I think about "AI takeover", I am imagining a very strong a smart AI, the one for which the success is more plausible. But before we get strong AIs, we will have weak AIs, so the first takeover attempts will be made by them. Maybe even the first successful takeover attempt.

A very strong and smart AI would however do thousand different things at the same time. Unlike the chess bot, which only plays on one chessboard, the AI could e.g. have separate plans to taking over each specific country. Many plans to take over one specific country would not ... (read more)

2Eli Tyre6d

I didn't know this about chess engines! If so, that does make fending off against a misaligned superintelligence seem even scarier!

Harjas Sandhu's Shortform

Harjas Sandhu9h3-6

Thought in progress: epistemic humility is not a substitute for actual humility (or professed humility). You only get to cry wolf once, but you can probably warn about potential wolves several times—so long as you don't burn goodwill on an incorrect or overconfident prediction.

I think epistemic humility helps to increase trust and confidence in EA/Less Wrong-type spaces, but I think professed humility is far more helpful when it comes to public-facing AI comms, particularly as scenarios get more intense and specific (e.g. prefacing AI doom predictions with... (read more)

152334H's Shortform

152334H21h10

I do not want, "Disciplines like psychology, philosophy, religious studies, and the social sciences [to] have an important role to play [...] in determining how AI systems develop and behave."

Showing 3 of 5 replies (Click to show all)

4Mitchell_Porter15h

Why not?

152334H10h10

I would prefer a future where AI models are not prescribed false frameworks of the human psyche, not predisposed to 'human vibe' philosophy, not innately desirous of any historical faith, nor credulous of the various dubious subsets of current social science.

I'm learning that common lesswrong readers do not think in this matter, but it is not clear to me in what direction. Is it due to a literalist interpretation of the OP, neglecting the contemporary context? Is it due to higher trust, affiliation, and support for the disciplines? Is it because readers tend to prefer anthropomorphic interpretations of AI behavior?

2Karl Krueger17h

Hmm. Maybe my question came across as ironic or accusatory or something? Sorry, it wasn't meant as such. Let me unpack it and pick some specific instances, and maybe we can find if there's a crux here. Philosophy includes ethics. Social sciences includes economics. If one doesn't want philosophy or social sciences to have a role in how AI systems develop and behave, that entails that one doesn't want AI systems to be affected by economics or ethics.

XelaP's Shortform

XelaP10h10

There's manifold markets for the LW review; presumably making them for curated status would not be worth it given how fast you'd need to be to be useful and how your net needs to be wider (maybe? alternatively "posts about x karma threshold in last 1-5 days not curated" seems pretty small if you set the threshold to what seems like my minds using in my heuristic).

But for example: it doesn't seem hard to tell in advance. e.g. i'd bet mana that gene smith's practical guide to superbabies gets curated by the end of the week. I wish I could take a curate bot'... (read more)

No77e's Shortform

No77e17h6-6

Rationalists and Pause AI people on X are accusing Davidad of suffering of AI psychosis. I think it's them who have lost the plot actually, not Davidad. The move here looks political, rather than truth-tracking. "Davidad is now my political opponent, so I'm accusing him of being crazy." This happened to Emmet Shear too at some point.

I also strongly believe AI psychosis to be a far more limited phenomenon than people here seem to believe. I think you're treating it as a good soldier in your army of arguments rather than investigating it truthfully for what it is.

Showing 3 of 4 replies (Click to show all)

2StanislavKrym11h

Most likely the OP means this quick take by Ivan Vendrov. Davidad ended up believing that the LLMs have been grokking the Natural Abstract Goodness which is unlikely to be capturable by existing benchmarks. While I do buy the idea that the NAG exists, I don't think that I understand how one can check that the LLMs really understood it.

DW1110h10

Davidad also announced the other day that he's leaving ARIA to pursue a research agenda focused on working with AIs on moral philosophy.

https://x.com/davidad/status/2039390998694891816

1DW1113h

I don't think Davidad has AI psychosis (his views seem to be quite coherent and aligned with his long-standing views on moral realism). But I think he is quirky and maybe expressing his views in a deliberately provocative way. He implied in his thread on leaving ARIA that it was his equivalent of the Death with Dignity post, which was also on April Fool's, and that thread definitely had some weird stuff (e.g. calling it "Alignment with Awakening").

Expertium's Shortform

Expertium13d*17-6

Do you know a person who believes that ASI will be created in <50 years who ISN'T in the LW/rationalists circle?

My parents don't believe that a superintelligent AI will be created within this century, or ever for that matter, or that AI will ever take jobs. My relatives laugh at the idea of AI solving a high school math problem and think state-of-the-art AI is on the level of GPT-2 (I mean that the capabilities they have in mind are on the level of GPT-2, not that they know what GPT-2 is). My friend who is an organic chemist ~~laughs at the idea of AI doi~~... (read more)

Showing 3 of 12 replies (Click to show all)

Viliam12h20

Do you know a person who regularly tries doing new things on a computer, and isn't somehow connected to the "TESCREAL" circle? (At least in the sense of "used to read sci-fi when young"?)

It is quite easy to underestimate what the LLMs can do, if you simple never use them, and only get your opinions from other people who never use them either.

9Carl Feynman11d

I've been in the mostly-academic AI circles in the Boston area for decades. Lots of people in these circles think ASI is plausibly close. I think it's difficult to pay close technical attention to the field and not think that AI is currently par-human, and improving every year. Many of them disagree with LessWrong concensus that it will be fatal, of course. Or simply haven't thought it through.

12Haiku12d

Most people in the PauseAI movement are not in the LW/rationalist circle. Some joined as rationalists or EAs, especially early on, but today most are normies (or were when they joined, anyway). I personally found LessWrong and the forecasting community through AI Safety, not the reverse. I now organize for PauseAI Phoenix, a local group of PauseAI US. I have face-to-face conversations on a regular basis with people who believe ASI will be created within 2-20 years unless we prevent that from happening.

Wei Dai's Shortform

Wei Dai1mo1326

Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. With very few exceptions, billionaires spend or give away their money haphazardly, philosophers don't bother to think about long term implications of AI on philosophy production (positive or negative), Terence Tao spends his time wireheading on abstract math instead of doing anything remotely like instrumental convergence. Unlike my youthful expectations (upon reading Vernor Vinge), there are no university departments filled with super-geniuses cha... (read more)

Showing 3 of 29 replies (Click to show all)

Mateusz Bagiński12h60

[Epistemic status: butterfly speculation, not confident about this, but I think it's an idea worth taking seriously.]

Terence Tao spends his time wireheading on abstract math

So I was initially [skeptical of]/[slightly repulsed by] this framing^[1], but after davidad's recent LLM-induced "awakening", I am starting to wonder that perhaps very high-g (+ high-NFC?) people tend to have a tendency towards something that is not very badly described as wirehead-y.^[2]

If you think about cognition/theorizing as divided into generation+verification, then we can take ver... (read more)

6Mateusz Bagiński20d

Why are you more uncertain about the consequences of Christianity being good than about those of Rome?

3Wei Dai22d

I'm having more trouble understanding this comment than the parent one. Can you please expand on this more? Note that I'm quite confused/uncertain about what morality is; see Six Plausible Meta-Ethical Alternatives for an explanation of this.

Praphull Kabtiyal's Shortform

Praphull Kabtiyal13d40

We have all heard the "AI just predicts the next word/token" and "AI just thought of X because it is in the training data" argument. I have a few ideas, first-draft stage, of experiments that might address this.

1) People invent artificial languages aka conlang (short for constructed language). The most famous examples being Esperanto, Klingon, and Tolkien's Elvish. Someone can invent a new conlang that didn't exist till today, and by extension wasn't present in any training data of any LLM, and explain the rules to an LLM (after the training mode has alrea... (read more)

Viliam12h20

AI can already reason about the application that it wrote for my yesterday, so I am already convinced that it is not merely looking for answers in a pre-existing database. (Even if many people had similar ideas before, they didn't use the same names for the objects.)

AI can communicate fluently in Esperanto, but there are already hundreds of books in digital form.

I have designed a puzzle game, and then AI successfully solved a few levels.

...so I don't need any more evidence. But could be useful for other people.

You don't have to invent a new language, it wo... (read more)

XelaP's Shortform

XelaP1d*210

For about a yearish I've been with varying frequency writing down things I've learned in a notebook. Partly this is so that I go "ugh I haven't even learned anything today, lemme go and meet my Learning Quota", which I find helpful (I don't think I'm goodharting it too much to be useful). Entries range from "somewhat neat theorem and proof in more detail than I should've written" to "high level overview of bigger subjects", or "list of keywords that I learned exist". For example, recently I learned that sonic black holes which trap phonons (aka lattice vib... (read more)

XelaP12h20

Also: I think I learned the fastest when I was in highschool, when there were both more low hanging fruit and I was spending much more time on it (unrelated to school except that I might've done less had I had more friends). And glacially slow before that.

So... perhaps this explains a bit more of the thing where I and felt like I became an adult mind a little after the start of that 'intelligence explosion'.