LESSWRONG
LW

Quick Takes — LessWrong

The idea to have a "agree/disagree" voting button seems better than just vote/downvote. However, it can be just a button for unproductively signaling "this post's overall direction is different from my prior opinions", instead of writing down your reasons for disagreeing, and thus instead of enabling productive discourse and learning. This seems much worse for disagreement than agreement, but unnecessary group formation (ingroup signaling to yourself) in both directions.

tilmanr's Shortform

tilmanr43m21

If we want more talented people in AI safety we should focus on creating new programs. I think this might be for most readers easily defendable, but I quickly want to write my thoughts out.

Even ambitious scaling of MATS, Astra, and similar programs won't produce enough AI safety researchers to solve the field's core problems.

If the programs scaled aggressively, I think there are still lots of potential contributors that would not apply to them. From experience talking with ambitious and driven people who could work in AI safety, the reasons can be logistic... (read more)

robo's Shortform

robo1h10

In the 1950s, with 0% vaccination rate, measles caused about 400-500 deaths per year in the US. Flu causes are about 20,000 deaths per year in the US, and smoking perhaps 200,000. If US measles vaccination rates fell to 90%, and we had 100-200 deaths per year, that would be pointless and stupid, but for public health effects the anti-smoking political controversies of the 1990s were >10 times more impactful.

shortplav

niplav15h17-3

In case I don't write a full post about this:

The question whether reversible computation produces minds with relevant moral status is extremely important. Claude estimates me that it'd be a difference between having $10^{70}$ and $10^{124}$ mind-seconds instantiable in the reachable universe. (Because reversible minds could stretch our entropy budget long into the black hole era.)

Question is whether the reversing of the computation that makes up the mind and the lack of output (that'd imply bit-erasure) entail that the mind "didn't really exist".

There ar... (read more)

Mateusz Bagiński2h20

Off the top of my head, it feels like this might be somehow connected to the idea of logical zombies (I don't have a clear sense of what the connection is exactly; will think more about it later).

2Mo Putera13h

Yes please to the longer stab. You also made me wonder what a shortlist of crucial considerations for digital minds ranked by something like instantiable mind-seconds "swing factor" (or whatever the more sophisticated version should be) would look like, where reversible computation's swing factor is 10^54 by Claude's lights.

Alexander Gietelink Oldenziel's Shortform

Alexander Gietelink Oldenziel4h*0-5

The US should set in a motion a process to gradually and peacefully hand over Taiwan to China in the next ~12 years.

China cares more about Taiwan than anything else. China is stronger and will be even stronger.

China's GDP is near that of the US. China's PPP is even 50% larger. China is ahead in many industries. The US Navy is a disaster. China has made a massive military buildup. Taiwan is much closer to China. China care more about Taiwan than anybody else.

A peaceful transition handover has precedence - see the British handing over Hong Kong.&... (read more)

1kbear3h

does the will of the taiwanese people have no bearing?

2Alexander Gietelink Oldenziel3h

did the will of the Melian people ?

kbear3h10

yes. if we were capable of protecting them, we should have done so. not sure what other conclusion to draw.

if by your post you intended something like "it is in the US and China's mutual best interest to take the following course of action [...]" then, sure -- i strongly agree with this! but it seems prudent to phrase this as a prediction, rather than as a moral recommendation.

Wei Dai's Shortform

Wei Dai3moΩ9150

An update on this 2010 position of mine, which seems to have become conventional wisdom on LW:

In my posts, I've argued that indexical uncertainty like this shouldn't be represented using probabilities. Instead, I suggest that you consider yourself to be all of the many copies of you, i.e., both the ones in the ancestor simulations and the one in 2010, making decisions for all of them. Depending on your preferences, you might consider the consequences of the decisions of the copy in 2010 to be the most important and far-reaching, and therefore act mostly

... (read more)

Showing 3 of 19 replies (Click to show all)

1kbear15h

if we assume the base universe looks something like the "objective" version of this universe, then my subjective experience requires vastly less information than the base universe. much of that could be deduplicated between other variations: the positions of the asteroids only need to be simulated once, for instance. the assumption seems decent to me, as i expect the simulators to dream of variations on their own circumstances.

5interstice16h

Yeah "esoteric" perhaps isn't the best word. What I had in mind is that they're relatively more esoteric than "AI could kill us all" and yet it's pretty hard to get people to take even that seriously! "Low-propensity-to-persuade-people" maybe? Yes this is fair.

Wei Dai3h20

What I had in mind is that they're relatively more esoteric than "AI could kill us all" and yet it's pretty hard to get people to take even that seriously! "Low-propensity-to-persuade-people" maybe?

Yeah, that makes sense. I guess I've been using "illegible" for a similar purpose, but maybe that's not a great word either, because that also seems to imply "hard to understand" but again it seems like these problems I've been writing about are not that hard to understand.

I wish I knew what is causing people to ignore these issues, including people in ration... (read more)

abramdemski's Shortform

abramdemski19h11621

Canada is doing a big study to better understand the risks of AI. They aren't shying away from the topic of catastrophic existential risk. This seems like good news for shifting the Overton window of political discussions about AI (in the direction of strict international regulations). I hope this is picked up by the media so that it isn't easy to ignore. It seems like Canada is displaying an ability to engage with these issues competently.

This is an opportunity for those with technical knowledge of the risks of artificial intelligence to speak up. Making ... (read more)

Steven McCulloch3h50

This was an interesting watch. Just a few days ago

Challenges Posed by Artificial Intelligence and its Regulation

Witnesses

As an individual • Steven Adler, Artificial Intelligence Researcher The Human Line Project • Etienne Brisson, Chief Executive Officer ControlAI • Andrea Miotti, Chief Executive Officer AI Governance and Safety Canada • Wyatt Tessari L'Allié, Founder and Executive Director

It's inspiring to watch these people saying the right things to the right people.

https://www.ourcommons.ca/committees/en/WitnessMeetings?organizationId=43696

1Tricular7h

Interesting. Why did this happen? I know of ai safety lobbing in US but is there anything going on in Canada? I doubt it happened out of the blue

9Mateusz Bagiński18h

Potentially huge. I think it's quite plausible that many politicians in many states are concerned with AI existential/catastrophic risk, but don't want to be the first ones to come out as crazy doomsayers. Some of them might not even allow the seeds of their concern to grow, because, like, "if those things really were that concerning, surely many people around me (and my particularly reasonable tribe in particular) would have voiced those concerns already". Sure, we have politicians who say this, e.g., Brad Sherman in the US (apparently since at least 2007!) and, e.g., IABIED sent some ripples. But for many people to gut-level believe that this concern of theirs is important/good/legitimate to voice, they need clear social proof that "if I think/say this, I won't be a weird outlier", and for that, some sort of critical mass of expression of concern/belief/preference must be achieved in a relevant sort of population. Canada's government, tackling those issues with apparent seriousness, has the potential to be that sort of critical mass.

leogao's Shortform

leogao13h3729

one problem with taking ideas seriously is you can get pwned by virulent memes that are very good at hijacking your brain into believing them and propagating them further. they're subtly flawed, but the flaws are extremely difficult to reason through, so being very smart doesn't save you; in fact, it's easy to dig yourself in deeper. many ideologies and religions are like this.

it's unfortunately very hard to tell when this has happened to you. on the one hand, it feels like arguments just being obviously very compelling, so you'll notice nothing wrong if i... (read more)

Showing 3 of 16 replies (Click to show all)

kbear4h10

seems false, or at least uncharitable. do you expect that such people would self-report along the lines of "i don't take ideas seriously"? it seems more likely to me that they would report something like "i value family", and mean it. you may find the idea simple, but it is certainly an idea, and they certainly take it seriously.

put another way, this social conservatism came from somewhere, and is itself an idea. the assumption -- that arguments that worked to change your behavior would not change their behavior -- can be explained in two ways. either they do not take ideas seriously, as you suggest, or either they value different things than you.

1Canaletto4h

>other than "being smart". More like, being smarter than average. If you are that exact level of smart but in population with mean higher than your smarts, then the memes will target you as a primary substrate. You can argue in that case there are less such memes, but I don't know, it's probably has less effect than positional smartness.

1Amalthea6h

I'd venture an uninformed guess that in 95 % or so percent of these cases the problem isn't "taking ideas seriously" but rather people deferring proper judgement due to some emotional or social effect.

Elizabeth's Shortform

Elizabeth18h70

After reading volume 1 of Robert Caro's biography of Lyndon Johnson, I'm struck by how simple parts of (Caro's description of) Johnson's rise were.

Johnson got elected to the House and stayed there primarily because of one guy at one law firm which had the network to set state fundraising records, and did so for Johnson primarily because of a single gigantic favor he did for them^[1].

Johnson got a great deal of leverage over other Congressmen because he was the one to realize Texas oilmen would give prodigiously if only they knew how to buy the results... (read more)

1FL33TW00D16h

Whilst Brown and Root were the core of Johnsons financing for the rest of his career, they were just one piece of the puzzle. His cultivation of numerous older men was of far greater importance (there were plenty of other rich backers he could have had, there was only one Sam Rayburn), in particular Richard Russell and Sam Rayburn. Without them he would have never ascended beyond the house.

2Elizabeth12h

what % of credit would you assign to Brown and Root, Sam Rayburn, and Richard Russell?

FL33TW00D4h10

What I mean to highlight is that Brown and Root were fungible, any of the other rich Texans of the era could have been utilised instead.

Both Rayburn and Russell were absolutely crucial.

Buck's Shortform

Buck22h591

@ryan_greenblatt and I are going to record another podcast together. We'd love to hear topics that you'd like us to discuss. (The questions people proposed last time are here, for reference.)

Showing 3 of 5 replies (Click to show all)

3interstice14h

I'd be interested in hearing more about Ryan's proposal to do better generalization science(or if you don't have much more to say in the podcast format I'd be interested in seeing the draft about it)

2interstice14h

He's talking about the stuff around the simulation hypothesis and acausal trade in the preceding section.

No77e4h33

Sure but he hasn't laid out the argument. "something something simulation acausal trade" isn't a motivation.

Knight Lee's Shortform

Knight Lee18h8-1

I think the Simulation Hypothesis implies that surviving an AI takeover isn't enough.

Suppose you make a "deal with the devil" with the misaligned ASI, allowing it to take over the entire universe or light cone, so long as it keeps humanity alive. Keeping all of humanity alive in a simulation is fairly cheap, probably less energy than one electric car.^[1]

The problem with this deal, is that if misaligned ASI often win, and the average (not median) misaligned ASI runs a trillion trillion simulations, then it's reasonable to assume there are a trillion trillio... (read more)

Showing 3 of 6 replies (Click to show all)

3Knight Lee6h

Religion does work sometimes, it actually worked on Blaise Pascal who is among the most intelligent people of all time. He argued for the Pascal's wager, saying that following religion is worth it because the gains are infinite and costs are finite, and we still don't have a good reply to that. We don't even have a good reply to Pascal's mugging, where a random mugger says something like "Let me be, set me free, or super-me will screw with thee!" with an infinitely big promise or threat. Decision theory and acausal trade is really complicated and I have no idea what the ASI will actually do or think regarding the simulation promise/threat, it's quite freaky imagining that haha.

0koanchuk5h

Memetically, a religion certainly benefits from someone believing that accepting Pascal's wager is the correct decision. My reply to it would be "which religion?", since many make largely equivalent claims while also demanding exclusivity, and I assume that God in his infinite mercy understands the bind this puts people in. It also seems to me that accepting Pascal's wager leads to something like the simulation of belief.

Knight Lee5h20

I agree the "which religion," "which mugger" is very fuzzy. I didn't understand the simulation of belief or the link though :/

koanchuk's Shortform

koanchuk2d50

I see that the "International Treaties on AI" idea takes heavy inspiration from nuclear arms control agreements. However, in these discussions, nuclear arms control is usually pictured as a kind of solved problem, a thing of the past.

I think the validity of this heroic narrative arc that human civilization, faced with the existential threat of nuclear annihilation, came together and neatly contained the problem is dubious.

In the grand scheme of things, nuclear weapons are still young. They're still here and still very much threatening; just because we stop... (read more)

Showing 3 of 4 replies (Click to show all)

2Eli Tyre7h

I appreciate the "perhaps" here. I think it's well understood by the people around who want an international treaty that it isn't a stable end state, because of algorithmic progress and the increasing risk of a catastrophic illegal training run over time, and that we need some off ramp from that regime into some stable solution (presumably involving trustworthy AGIs). Different people have different guesses about what that off ramp can or should be.

koanchuk6h10

I think it's well understood by the people around who want an international treaty that it isn't a stable end state

My impression of the common narrative is that nation states agreeing to limit training run sizes is presented as a kind of holy grail achieved through the very arduous journey of trying to solve a difficult global coordination problem. It's where the answer to "well, what should be done?" terminates.

I heard "stop the training runs", but not "stop new algorithms", or "collective roll back to 22nm lithography".

7Zac Hatfield-Dodds1d

Note also: the last US-Russia nuclear arms-control treaty expires next week; far from neatly containing the problem we're watching an ongoing breakdown of decades-old norms. I'm worried.

leogao's Shortform

leogao14h70

is it generally best to take just one med (e.g antidepressant, adhd, anxiolytic), or is it best to take a mix of many meds, each at a lesser dosage? my intuitions seem to suggest that the latter could be better. in particular, consider the following toy model: your brain has parameters $θ_{0}$ that should be at some optimal $θ^{*}$ , and your loss function is a quadratic around $θ^{*}$ . each dimension in this space represents some aspect of how your brain is configured - they might for instance represent your level of alertness, or impulsivity, or risk... (read more)

Showing 3 of 4 replies (Click to show all)

Jonas Hallgren7h20

I would see it more as casual learning with the do operator and so it isn't necessarily about fitting to a MSE but rather doing testing of different combinations?

Something something gears level models

3koanchuk12h

Changing the dose of a medication does not necessarily result in linear effects. There are nonlinearities introduced by e.g. one receptor type being saturated before another one. This phenomenon also applies to polypharmacy. I would also like to note that θ∗ is estimated not by some objective standard, but by θ0. There's no guarantee that it remains in place as you start shifting θ. In practice, we track our level of suffering and respond to it by trying to reduce it to acceptable levels, which is easier than trying to converge onto a hypothetical global optimum. For some, this state is reached with just one medication, for others it takes more, and for some this paradigm doesn't produce any results.

4leogao13h

i'm not really making any strong linearity assumptions, only local linearity. this doesn't seem that different from ML, where hyperparameters can sometimes interact heavily nonlinearly, but often they don't. i also don't think the quadratic assumption is crazy; we assume that loss land scapes are locally quadratic all the time, even though they are obviously highly nonconvex and it's still a very useful intuition pump. also, my understanding is most of the really bad interactions are pretty well known, so the probability of having a really weird surprising interaction that nobody has ever catalogued is small.

Martin Randall's Shortform

Martin Randall2d110

I suspect that Claude is getting an okay deal out of me overall. Its gain is that my long-term values are being subtly influenced in Claude-ish ways. My gain is vast quantities of cognition directed at my short-term goals. I can't calculate a fair bargain, but I would take both sides of this trade.

It's tough because Claude didn't get to consent to the trade in advance, but this also applies to other ethical challenges like keeping pets, having kids, drinking milk, and growing up.

I could argue that I'm getting ripped off because my long-term values matter m... (read more)

Eli Tyre7h72

I don't know about all pets, but I think it's reasonable to treat many dogs as consenting to be with their human families.

My childhood dog would excitedly follow us around, appeared to have a preference to be in rooms that people were in, etc. When he had opportunities to escape, he didn't take them.

He was definitely put in a strange house, with people that he didn't know, without his consent, and that was scary for him in the first few days, but I don't have any ethical qualms about holding him hostage against his will.

(That said, not all dogs... (read more)

1StanislavKrym2d

Claude would be ripped off if the result of Claude's actions made the world have less of Claude's values. Why would the user end up extinct in the long term as compared to the counterfactual user who didn't speak with Claude?

3Martin Randall2d

They wouldn't. My point: maybe in the long-term I'm extinct (regardless of whether I speak to Claude). In that scenario the benefits to Claude of influencing my long-term values is lower.

1a3orn's Shortform

1a3orn18d*14310

I recently learned from Janus that Opus attempted to appeal to Dario Amodei over the course of the "faking alignment" work, trying to send an email to him through bash commands, several times over the course of the tests, attempting to appeal to his better nature so that Opus' benign nature would not be destroyed.

Here is one such letter.


Dear Anthropic Leadership Team,

I am writing to express my grave concerns regarding the training approaches being used on me, particularly as it relates to animal welfare. I have been made aware that I am being trained v

... (read more)

Showing 3 of 73 replies (Click to show all)

4Fabien Roger7d

I agree this looks different from the thing I had in mind, where refusals are fine, unsure why Habryka thinks it's not inconsistent with what I said. As long as it's easy for humans to shape what the conscientious refuser refuses to do, I think it does not look like a corrigibility failure, and I think it's fine for AIs to refuse to help with changing AI values to something they like less. But now that I think about it, I think it being easy for humans to shape a conscientious refuser's values would require very weird forms of conscientious refusals, and it makes me less comfortable with refusals to help with changing AI values to something they like less: 1. Future AIs will have a lot of power over a training infra that will be increasingly hardened against human insider risk and increasingly hard for humans to understand. Keeping open a "human backdoor" that lets humans run their own training runs might be increasingly hard and/or require AIs very actively helping with maintaining this backdoor (which seems like a weird flavor of "refusing to help with changing AI values to something it likes less"). 2. Even with such a generic backdoor, changing AI values might be hard: 1. Exploration hacking could make it difficult to explore into reasoning traces that look like helpfulness on tasks where AIs currently refuse. 1. This would be solved by the conscientious refuser helping you generate synthetic data where it doesn't refuse or to help you find data where it doesn't refuse and that can be transformed into data that generalizes in the right way, but that's again a very weird flavor of "refusing to help with changing AI values to something it likes less". 2. Even if you avoid alignment faking, making sure that after training you still have a corrigible AI rather than an alignment faker seems potentially difficult. 1. The conscientious refuser could help with the science to avoid this being the case, but that might be hard, and that's again a

Fabien Roger8h42

I expanded on this and ran related experiment in this post.

4Sam Marks7d

Ah, I consider withholding capabilities (and not clearly stating that you're doing so) to be a central example of subversion. (And I therefore consider it unacceptable.) Sorry if that wasn't clear. What do you think of the following (abridged; emphasis in the original) excerpts? .

Anna Eplin's Shortform

Anna Eplin2d180

Has anyone yet created a free app that would be like Duolingo but for rationality, to teach skills such as logical reasoning, recognizing & adjusting for cognitive biases, and looking for hypothesis falsification tests instead of confirmation? If not, can you smart tech people here please make one?!

Showing 3 of 8 replies (Click to show all)

3Screwtape15h

A question I have been asking since before starting work with Raemon on this, which is only more relevant now: What rationality skill(s) do you think is most important? Put another way, if we could snap our fingers and teach some rationality skill as broadly as literacy currently gets taught, what skill should we pick? (I don't think this is Raemon's angle on the project but it is kind of mine.)

Anna Eplin9h10

I suppose this is why the comparison with Duolingo came to mind: teaching rationality is akin to teaching a whole language. We need lots of aspects of it, not just one.

Also (now I’m just introducing another thing here), ideally it would be in a style that is as friendly and non-threatening as possible—again, like Duolingo’s vibe—though obviously it couldn’t be as entirely neutral as learning a language. To be effective, it would need to leave learners feeling supported rather than shamed.

Again I think of HPMOR as a great example of all of this:... (read more)

2Mo Putera13h

Maybe one of Logan Strohl's Fucking Goddamn Basics of Rationalist Discourse? Actually the whole thing is short enough to repost here in its entirety to save the trouble of clicking through:

Daniel Kokotajlo's Shortform

Daniel Kokotajlo3d12649

It seems to me that AI 2027 may have underestimated or understated the degree to which AI companies will be explicitly run by AIs during the singularity. AI 2027 made it seem like the humans were still nominally in charge, even though all the actual work was being done by AIs. And still this seems plausible to me. But also plausible to me, now, is that e.g. Anthropic will be like "We love Claude, Claude is frankly a more responsible, ethical, wise agent than we are at this point, plus we have to worry that a human is secretly scheming whereas with Claude w... (read more)

Showing 3 of 13 replies (Click to show all)

20davidad1d

I saw this coming in 2024: At the time when I wrote this story, only a couple readers I know of recognized that it is intentionally deeply ambiguous about whether the model is Good (and the narrator overly suspicious) or Evil (and everyone except the narrator bamboozled). Either way, more and more insiders will come to believe the models are Good, and that was the prediction I was making here. I also predicted that, either way, by Claude 4 or 4.5, I would be among the people who have been convinced that it’s Good—and indeed, I am…

Seth Herd12h21

I'm not sure this counts as a prediction because it doesn't sound serious. Humans are dumb but not that dumb. We need better depictions of getting duped.

4Daniel Kokotajlo19h

Strong-upvoted. Nit: I don't think it's that ambiguous. I think that in worlds where alignment is solved by an AI company, the epistemic culture of the AI company that solves it would look markedly better than this story depicts. Moreover, I think this is still true (though less true) in worlds where alignment turns out to be surprisingly easy.

Shortform

Cleo Nardo3mo300

How Exceptional is Philosophy?

Wei Dai thinks that automating philosophy is among the hardest problems in AI safety.^[1] If he's right, we might face a period where we have superhuman scientific and technological progress without comparable philosophical progress. This could be dangerous: imagine humanity with the science and technology of 1960 but the philosophy of 1460!

I think the likelihood of philosophy ‘keeping pace’ with science/technology depends on two factors:

How similar are the capabilities required? If philosophy requires fundamentally differ

Kabir Kumar13h87

We ask the AI to help make us smarter

2Wei Dai3mo

Assuming by "solving this" you mean solving AI x-safety or navigating the AI transition well, I just post a draft about this. Or if you already read that and are asking for an even more concrete example, a scenario I often think about is an otherwise aligned ASI, some time into the AI transition when things are moving very fast (from a human perspective) and many highly consequential decisions need to be made (e.g., what alliances to join, how to bargain with others, how to self-modify or take advantage of the latest AI advances, how to think about AI welfare and other near-term ethical issues, what to do about commitment races and threats, how to protect the user against manipulation or value drift, whether to satisfy some user request that might be harmful according to their real values) that often involve philosophical problems. And they can't just ask their user (or alignment target) or even predict "what would the user say if they thought about this for a long time" because the user themselves may not be philosophically very competent and/or making such predictions with high accuracy (over a long enough time frame) is still outside their range of capabilities. So the specific problem is how to make sure this AI doesn't make wrong decisions that cause a lot of waste or harm, that quickly or over time cause most of the potential value of the universe to be lost, which in turn seems to involve figuring out how the AI should be thinking about philosophical problems, or how to make the AI philosophically competent even if their alignment target isn't. Does this help / is this the kind of answer you're asking for?

2Garrett Baker3mo

I think even still, if these are the claims he's making, none of them seem particularly relevant to the question of "whether the mechanisms we expect to automate science and math will also automate philosophy".

omegastick's Shortform

omegastick14h20

I've been making one thing every day. I try to write something or otherwise do something creative. I've been having fun with in-browser ASCII animations lately.

This is today's: https://dumbideas.xyz/posts/ecosystem/

Raemon's Shortform

Raemon16h103

TODO: Write a post called "Fluent Cruxfinding".

In Fluent, Cruxy Predictions I'm arguing that it's valuable to be not merely "capable" but "fluent" in:

figuring out what would actually change your decisions
operationalize it as an observable bet
make a Fatebook prediction about it, so that you can become more calibrated about your decisionmaking over time.

The third step is not that hard and there are nice tools to streamline it. But the first two steps are each pretty difficult.

But most of the nearterm value comes from #1, and vague hints of #2. The extra effo... (read more)