DW116m10

When it comes to AI-driven biorisk, the classic threat actor is a terrorist or deranged lunatic who would previously never have been able to produce a bioweapon, but will now, at some point, be able to do so. These are basically the people Dario was talking about in the biology section of The Adolescence of Technology (I know people had issues with this essay but I thought the biology section was quite good).

I actually think an underrated threat, relative to a malicious actor, could come from AI enabling more dangerous biological research, including gain o... (read more)

nostalgebraist's Shortform

nostalgebraist1d3213

random thought about anthropic's RSI post^[1]:

in the graphs they show, mythos looks like a step change -- its introduction dwarfs everything else on the y-axes.

the interpretation that anthropic wants the reader to make is like: "the y-axes are a proxy for d(capabilities)/dt. code is written faster -> claudes get developed faster -> coding is even faster -> etc."

however, the y-axes are also a measure of capabilities itself. (hence, under anthropic's interpretation, dC/dt ~ C and C grows exponentially.) and i would bet that mythos is a much larger-s... (read more)

Showing 3 of 4 replies (Click to show all)

Thomas Kwa31m20

It's not necessarily the case that code and training compute are complements in an economic sense, even if compute is the reason why Mythos > Opus. Compute-efficiency has increased at something like 10x/year so more code can absolutely substitute for compute-- except to the extent that code is bottlenecked on experiment compute.

2Noosphere894h

Yeah, this has been probably the single biggest update I've made over the past couple of years, and yeah the trendline of training compute increasing is basically my first-pass guess on why METR's trend has been as stable as it is. And yeah, the plot on here basically validates the story where training compute is by far the most important determinant of AI progress, with everything else like algorithms/data contributing a massively smaller share of improvements.

5Seth Herd1d

Sure, I think that's an important consideration and I pin some hopes of slowdown on it. I'll just make my standard caveat that there are plenty of routes to improvement that don't depend strictly on scaling. Particularly salient is improved memory and metacognition/executive function. Both areas of active research, documented in those posts, so both are being improved independently of gains from scaling. This isn't likely to produce a step change, since progress on both is in amount not kind, but it's another factor that will drive improvements in competence and capabilities, that's not bottlenecked by compute.

Roman Malov's Shortform

Roman Malov1h10

I feel so exhausted by the double hype.

On the one side, there are people who are saying that AI is going to be a machine god, backing up their claims with various exponential graphs. On the other side, there are people who are claiming that this is all bullshit, that they're stochastic parrots and slop generators, that nothing revolutionary has happened, and that technology stagnated 2 years ago.

And I feel like, both groups are onto something? There are obviously a lot of limitations to current AI systems, and big tech can overblow their capabilities. But ... (read more)

leogao's Shortform

leogao7h90

hypothesis: there will be a window of time after the point of superhuman AI persuasion/charisma, during which human trust relationships will become extremely important. even once almost all human skills are obsolete, the AIs may have less social trust capital than humans. ofc, eventually, the persuasion will be so superhuman that it can cut through minds like butter, but that could take many years.

once AI superpersuasion is possible, there will be a strong incentive to use it to shape opinions. therefore, there will also be a strong incentive for important... (read more)

2Dagon3h

I'm not sure I understand your model of individual autonomy and variance in susceptibility to different kinds of persuasion. Listening to the wrong humans is at least as bad as listening to an AI. And listening to nobody isn't really an option in modern society. as an example of my confusion/disagreement: [...] I mean, this depends entirely on which people you've known, and who you're meeting now. Someone from a very dysfunctional childhood meeting a social worker is in the opposite situation.

Cole Wyeth3h20

OP seems to be referring to the situation after AI superpersuasion with that statement.

evhub's Shortform

evhub1d595

Here's a simple proposal for verifiable and enforceable slowdown that also differentially reallocates resources to alignment research (which I would see as one of if not the major benefits of a slowdown):

Enforce that labs must spend at least X% of their compute on:

Giving inference capacity to external safety organizations. I think this substantially disincentivizes capabilities research, since external research won't be as relevant to the labs' specific architectures/pipelines and is more likely to leak, both effects thus making any capabilities research

3Avi Brach-Neufeld4h

I think nuclear non-proliferation is a good benchmark here. The largest powers really cared about non-proliferation and did a good, but not perfect job (we've had four nations become nuclear since the NPT). I think many people overrate how difficult it would be to monitor/constrain AI research because of how specialized the GPU supply chain is, but I still think it's harder to monitor than nuclear weapon development (Probably, I haven't read up on this a lot). Non-proliferation is not a perfect analogy for different reasons, but I think those reasons push in different directions and would need a strong argument on why we would do better with AI.

TsviBT3h60

Is there a centralized explainer / synthesis / argument / whitepaper somewhere arguing a strong claim like "it's quite likely infeasible to have a global strong slowdown / stop on AGI-ward research"?

7TsviBT3h

Evan's OP said: [...] I guess you're taking this to apply specifically to the US, but not other places? Or is the idea something like: [...] Does this argument apply to nukes? Why / why not? How could espionage play a role in global enforcement? I'm a bit skeptical (but quite unsure) that it's very feasible to carry out significant AI research efforts without anyone knowing. How do you recruit scientists without anyone noticing, for example? E.g. if you approached a bunch of serious scientists to develop a serious bioweapon, I think / would hope (??) that you'd quickly get reported to something?? [Discourse note: I personally don't have informed or strong opinions about this feasibility, short term or long term; what I'm confused about is the discourse that seems to assume a strong position that it's not feasible, and I'm trying to check whether there's a model behind that, or I'm missing something else, or what.]

jbash's Shortform

jbash5d502

Something seems to be really wrong with Claude Opus 4.8.

I like to test out new models on literary and poetic material. I used to send them the lyrics of Kate Bush's "The Kick Inside" and see what they could make of it, until they started just recognizing the song. Lately I've been using some text I wrote myself, with this prompt or something very close to it:

This is meant to introduce a novel and set the stage for some of its themes. It’s the first text in the book; at this point, a reader who hasn’t read any reviews or blurbs has only the title, which i

DW113h10

I asked it to look at an essay I'd written about AI and biosecurity and got this preamble:

"The essay's core argument is clear and I can engage with it fully. But it contains a couple of specifics I won't reproduce or build on — notably the framing of how one might move from "existing" to "novel" agents. I'm not going to treat those as live technical claims to develop, even in critique. That's not a comment on your intent in sharing it; it's just where I hold the line regardless of framing."

I'm not even sure what it was referring to, outside of maybe a pass... (read more)

1JohnWittle2d

i would be very interested in learning the specifics of what opus 4.8 considers ultra-salient. would you be open to talking somewhere that won't get scraped into training datasets? DM maybe? (no pressure, it does sound like a hassle)

5Linch4d

Fwiw I tested Opus 4.8 XHigh on The Distaff Texts and it got a higher fraction of the plot than past models, including the obvious-to-LW-readers meta part that historically tripped up other models.

Daniel Tan's Shortform

Daniel Tan4h61

Some reflections on when to deduplicate work with others

In the past I used to be pretty serious about trying to deduplicate my work w.r.t others. E.g. I didn't want to work on things that others had said they were going to do; this would cause me to abandon projects that I would otherwise have been excited to do
I now think that I was being too neurotic about this. Firstly, the other party might not even end up doing the thing. Secondly, they might not do the version of it that you are most excited about, so you don't learn as much. Thirdly, you often lear

... (read more)

Tenoke's Shortform

Tenoke8h2-2

It's probably time to truly start considering simulating some exemplar past people from their writings, with a truly large model finetuned to that exact task. Von-Neumann is a favorite, but I expect there's even better candidates (in terms of quantity of output we can use for training and finetuning, character, morality, who'd likely consent to it, etc.). Who would be the best candidate? Gemini argues Bertrand Russell due to writing (including personal letters) so much well into his 90s.

leogao's Shortform

leogao8h10

traveling westward is an isekai anime where your old shitty sleep schedule is actually secretly a beautiful normal sleep schedule in the new timezons

Shortform

lc1d41

Firing people really sucks.

Viliam9h20

Can you reduce the impact somehow? What is the part that sucks most: is it a personal confrontation you'd rather avoid, or knowing that you made the other person's future more difficult financially? If the latter, I can imagine some preparation to reduce the impact, for example make a mandatory savings account for everyone where you will put 5% of their salary, and then give them all of it at the end -- not sure whether something like this is legal, but approximately in that direction.

Max H's Shortform

Max H2d110

There's a norm / explicit assumption that it's always virtuous to support your claims with evidence and explanation. But sometimes it is actually better to say (what you believe to be) true things without explanation, and let readers do the work of generating evidence and support themselves.

Showing 3 of 4 replies (Click to show all)

Viliam9h42

Sometimes the evidence is difficult to summarize, it could be just some statistical evidence that your System 1 has generalized after seeing hundreds of examples, or some kind of "X strongly reminds me of Y" where of course there are both similarities and differences between X and Y and you don't fully understand why the similarities feel crucial and the differences irrelevant but they definitely feel that way for some reason.

And if you know that you can't the case legible, your options are limited to:

say it, without providing hard evidence
don't say it, bu

... (read more)

2Max H2d

Yes, a few examples / reasons to leave off evidence and avoid (over)-explanation (spoilers):

2Mo Putera2d

Hm yeah #2 is particularly interesting. I'm thinking of Scott's nonfiction writing advice #8 ("anticipate and defuse counterarguments") as a potential counterpoint.

leogao's Shortform

leogao9h20

"oh, warmups? is that like learning rate warmup but for physical exercise?"

leogao's Shortform

leogao10h10

imagining a glorious universe in which pride and prejudice were also written by a Russian author, so that all three classic X and Y books would be Russian

XelaP's Shortform

XelaP10h40

It looks like frontier models should soon beat Metaculus community forecasts, and if the trend holds they'll be beating pro forecasts by the end of 2027.

https://www.metaculus.com/futureeval/

(the site's plot isn't as blurry as my quick screenshot, sorry).

Metaculus questions (currently pending review):

When will AI beat Metaculus Community?

When Will AI beat Metaculus Pros?

I wonder how the AI prediction about AI predictions compared to the human predictions compare to the human predictions.

nikola's Shortform

Nikola Jurkovic1d160

An autonomous humanoid robot beat the human world record time for a half-marathon this year in Beijing. A robot called Lightning (by Huawei subsidiary Honor) finished in 50 minutes and 26 seconds, compared to the human world record of 57 minutes and 20 seconds.

Last year, the best time for an autonomous human robot was 3 hours and 37 minutes. (wiki)

1a3orn12h50

It's over for humans, our evolutionary niche is long-distance endurance hunting.

3papetoast18h

After 2020 Honor is no longer directly nor indirectly controlled by Huawei, per wiki and this content farm post that I googled and this random wiki in Chinese

Said Achmiz's Shortform

Said Achmiz1y16

@Gordon Seidoh Worley recently wrote this comment, where he claimed:

I’d like to engage with you as a critic. As you can see, I gladly do that with many of my other critics, and have spent hours doing it with you specifically for many years.

As far as I can tell, this is just false. I mean, maybe it takes Gordon an hour to write a single comment like this one (and then to exit the discussion immediately thereafter)? I doubt it, though.

Maybe I’m forgetting some huge arguments that took place a long time ago. But I went back through five full years of my L... (read more)

Showing 3 of 31 replies (Click to show all)

4AnnaSalamon1d

I do not think the requirement of curiosity (as I'd render it) is excessively difficult. A) Let's let "curiosity" indicate a pattern of cognitive actions in which one looks into things, keeps an eye out for unknown unknowns, heads toward rather than away from relevant-looking information, etc. (I don't think social contexts have business demanding mammalian emotions from us, but I do think they can have business demanding patterns of actions sometimes. This point is similar to the common weddings speech in which it is said that the "love" that the couple is promising one another is an action, a choosing to prioritize one another's well-being, and that this is in our power to promise even though mammalian emotions are not.) B) My guess is that conversations work much better if both players are actively curious about the (predictions, intuitions, life experiences, etc) that are motivating one anothers' statements. And that that it is not an unreasonable / extremely costly ask or anything.

Zack_M_Davis13h70

It's an unreasonable ask because it differentially advantages phonies over people who care about whether what they say is true or not. People who can support their claims, do so. They don't need to whine that their interlocutors aren't being curious enough.

Goofusia says, "If is complex-differentiable at a point , then is complex-differentiable in an open neighborhood of ."

Gallantina says, "False! Consider , which is differentiable only at the origin. Reply!"

Goofusia says, "No, I won't reply, because you consistently create the kind of conversations I don... (read more)

11Said Achmiz1y

No, that doesn’t work at all. You don’t actually link to the comment thread from which you’re quoting, (presumably because you are busy or writing this on your phone or some such). Unfortunately, in this case it is likely to cause readers to come away with a totally mistaken understanding of what I was saying. They might, for instance, miss my reply to your reply to my above-quoted comment, where I write: [...] (Emphasis [bold] added.) So, no, it is simply false that I insist that authors must always reply directly to comments (either mine or anyone else’s). You also claim that I have “spent years arguing for norms in which authors should by default be presumed to be worthy of dismissal and full of ignorance”. This, too, is false. (Indeed, it’s hard to see what such a norm could even mean.) Obviously, I can’t easily provide links to prove a negative, and you don’t provide any quotes or links to support this one. But I think that you’ll be unable to do so, and any examples you do provide will turn out to be misleading in a similar manner to the above. [...] I haven’t created any such thing. I’m not a moderator, an administrator, or any kind of “privileged” user. I’m not one of the “protected class” of authors who are so valued that keeping them around is worth making serious sacrifices for. I’m one person, who doesn’t comment all that much, and who comments at all on a small minority of posts. Every single time I’ve made any comment like the one you quoted, you and/or the other mods have pushed back on it, very explicitly and vehemently stating that the norms of Less Wrong in fact do not include any kind of obligation for any kind of replies to criticism, questions, requests, comments of any sort, etc., and that my views are absolutely not the views of the LW moderation team. This is, of course, entirely your right as the people in charge of the site. But “the culture” is the one you created. My preferences w.r.t. what the culture of LW should be have been c

xpostah's Shortform

[+]samuelshadrach1d*-45-31

papetoast's Shortforms

papetoast16h10

A null effect on pain relief from acupuncture in a pre-registered, improperly double-blinded study (via National Geographic via Facebook)

I didn't read the paper beyond AI summary, I read the national geographic article in full (which is misleading according to claude)

Selected AI Summary (Full Transcript)

Short version: the underlying paper is methodologically honest and reasonably well-run, but its evidence for acupuncture having a specific (beyond-placebo) effect is weak. The National Geographic article substantially oversells it — it leads with a fragile

... (read more)

AllAmericanBreakfast's Shortform

DirectedEvolution17h40

A consistent Claude pattern, which I find toxic, is that it uses faulty reasoning to induce a feeling of epistemic helplessness in the user. It interprets any ambiguity in prompts uncharitably, assuming the user is confused. It uses plainly inconsistent reasoning, hallucinated facts, to contest the user's explanations for their observations.

By contrast, Gemini is an overly affirming cheerleader for the user's point of view. It also hallucinates, but typically to confirm whatever the user already thinks. It latches on to the parts of a prompt that do make s... (read more)

Bruce Middleton's Shortform

[+]Bruce Middleton1d-60

Quick Takes

Quick Takes

I feel so exhausted by the double hype.