LESSWRONG
LW

Thomas Kwa's Shortform — LessWrong

Thomas Kwa's Shortform

22nd Mar 2020

1 min read

11 Ω 7

This is a special post for quick takes by Thomas Kwa. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Thomas Kwa's Shortform

2Bogdan Ionut Cirstea

2Alex K. Chen (parrot)

-1the gears to ascension

26the gears to ascension

5Bogdan Ionut Cirstea

29Alexander Gietelink Oldenziel

2Alexander Gietelink Oldenziel

2Bogdan Ionut Cirstea

35Thomas Kwa

9adamShimi

2Alexander Gietelink Oldenziel

6Alexander Gietelink Oldenziel

0the gears to ascension

4Alexander Gietelink Oldenziel

4Noosphere89

4Alexander Gietelink Oldenziel

4Noosphere89

2Alexander Gietelink Oldenziel

4Stephen Fowler

4Bogdan Ionut Cirstea

6Alexander Gietelink Oldenziel

2Alexander Gietelink Oldenziel

2Alexander Gietelink Oldenziel

-1the gears to ascension

359 comments, sorted by

top scoring

Click to highlight new comments since: Today at 3:22 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Thomas Kwa8mo*Ω411238

Edit: Full post here with 9 domains and updated conclusions!

Cross-domain time horizon:

We know AI time horizons (human time-to-complete at which a model has a 50% success rate) on software tasks are currently ~1.5hr and doubling every 4-7 months, but what about other domains? Here's a preliminary result comparing METR's task suite (orange line) to benchmarks in other domains, all of which have some kind of grounding in human data:

Observations

Time horizons on agentic computer use (OSWorld) is ~100x shorter than other domains. Domains like Tesla self-driving (tesla_fsd), scientific knowledge (gpqa), and math contests (aime), video understanding (video_mme), and software (hcast_r_s) all have roughly similar horizons.
- My guess is this means models are good at taking in information from a long context but bad at acting coherently. Most work requires agency like OSWorld, which may be why AIs can't do the average real-world 1-hour task yet.
- There are likely other domains that fall outside this cluster; these are just the five I examined
- Note the original version had a unit conversion error that gave 60x too high horizons for video_mme; this has been fixed (thanks @ryan_greenblatt )
Rate

... (read more)

[-]ryan_greenblatt8moΩ11172

but bad at acting coherently. Most work requires agency like OSWorld, which may be why AIs can't do the average real-world 1-hour task yet.

I'd have guessed that poor performance on OSWorld is mostly due to poor vision and mouse manipulation skills, rather than insufficient ability to act coherantly.

I'd guess that typical self-contained 1-hour task (as in, a human professional could do it in 1 hour with no prior context except context about the general field) also often require vision or non-text computer interaction and if they don't, I bet the AIs actually do pretty well.

[-]ryan_greenblatt8moΩ9113

I'm skeptical and/or confused about the video MME results:

You show Gemini 2.5 Pro's horizon length as ~5000 minutes or 80 hours. However, the longest videos in the benchmark are 1 hour long (in the long category they range from 30 min to 1 hr). Presumably you're trying to back out the 50% horizon length using some assumptions and then because Gemini 2.5 Pro's performance is 85%, you back out a 80-160x multiplier on the horizon length! This feels wrong/dubious to me if it is what you are doing.
Based on how long these horizon lengths are, I'm guessing you assumed that answering a question about a 1 hour long video takes a human 1 hr. This seems very wrong to me. I'd bet humans can typically answer these questions much faster by panning through the video looking for where the question might be answered and then looking at just that part. Minimally, you can sometimes answer the question by skimming the transcript and it should be possible to watch at 2x/3x speed. I'd guess the 1 hour video tasks take more like 5-10 min for a well practiced human, and I wouldn't be surprised by much shorter.
For this benchmark, (M)LLM performance seemingly doesn't vary much with video duration, invalidating that horizon length (at least horizon length based on video length) is a good measure on this dataset!

4Thomas Kwa8mo

There was a unit conversion mistake, it should have been 80 minutes. Now fixed. Besides that, I agree with everything here; these will all be fixed in the final blog post. I already looked at one of the 30m-1h questions and it appeared to be doable in ~3 minutes with the ability to ctrl-f transcripts but would take longer without transcripts, unknown how long. In the next version I will probably use the no-captions AI numbers and measure myself without captions to get a rough video speed multiplier, then possibly do better stats that separate out domains with strong human-time-dependent difficulty from domains without (like this and SWE-Lancer).

5ryan_greenblatt8mo

No captions feels very unnatural because both llms and humans could first apply relatively dumb speech to text tools.

[-]Thomas Kwa7moΩ2100

New graph with better data, formatting still wonky though. Colleagues say it reminds them of a subway map.

With individual question data from Epoch, and making an adjustment for human success rate (adjusted task length = avg human time / human success rate), AIME looks closer to the others, and it's clear that GPQA Diamond has saturated.

[-]Ted Sanders8mo102

Can you explain what a point on this graph means? Like, if I see Gemini 2.5 Pro Experimental at 110 minutes on GPQA, what does that mean? It takes an average bio+chem+physics PhD 110 minutes to get a score as high as Gemini 2.5 Pro Experimental?

2Thomas Kwa8mo

There is a decreasing curve of Gemini success probability vs average human time on questions in the benchmark, and the curve intersects 50% at roughly 110 minutes. Basically it's trying to measure the same quantity as the original paper (https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) but the numbers are less accurate since we have less data for these benchmarks.

9Daniel Kokotajlo8mo

I wish I had thought to blind myself to these results and try to predict them in advance. I think I would have predicted that Tesla self-driving would be the slowest and that aime would be the fastest. Not confident though. (Solving difficult math problems is just about the easiest long-horizon task to train for,* and in the last few months we've seen OpenAI especially put a lot of effort into training this.) *Only tokens, no images. Also no need for tools/plugins to the internet or some code or game environment. Also you have ground-truth access to the answers, it's impossible to reward hack.

[-]gwern8moΩ8243

I think I would have predicted that Tesla self-driving would be the slowest

For graphs like these, it obviously isn't important how the worst or mediocre competitors are doing, but the best one. It doesn't matter who's #5. Tesla self-driving is a longstanding, notorious failure. (And apparently is continuing to be a failure, as they continue to walk back the much-touted Cybertaxi launch, which keeps shrinking like a snowman in hell, now down to a few invited users in a heavily-mapped area with teleop.)

I'd be much more interested in Waymo numbers, as that is closer to SOTA, and they have been ramping up miles & cities.

5Thomas Kwa8mo

I would love to have Waymo data. It looks like it's only available since September 2024 so I'll still need to use Tesla for the earlier period. More critically they don't publish disengagement data, only crash/injury. There are Waymo claims of things like 1 disengagement every 17,000 miles but I don't believe them without a precise definition for what this number represents.

8lemonhope8mo

You could add cooking tasks with robots.

4StanislavKrym8mo

For some reason, all current benchmarks, with the sole exception of OSWorld[1], now seem to differ by a factor of less than 3. Does this imply that the progress in every benchmark is likely to slow down? 1. ^ OSWorld resembles a physical task, which LLMs tend to fail. However, the article about LLMs failing basic physical tasks was written in April 14, before the pre-release of Gemini Diffusion. Mankind has yet to determine how well diffusion-based LLMs deal with physical tasks.

4Unnamed8mo

The 'regression to the mean' pattern is striking: domains with a lower starting point have been growing faster, and those that started with a longer horizon have mostly been grower slower. I wonder if that pattern of catchup growth & drag on leaders will mostly hold up over more time and with a larger set of task types.

3eggsyntax8mo

Just to be sure: as in the METR results, 'horizon' here means 'the time needed to complete the task for humans with appropriate expertise', correct? I assume so but it would be useful to make that explicit (especially since many people who skimmed the METR results initially got the impression that it was 'the time needed for the model to complete the task').

2Bogdan Ionut Cirstea7mo

I would love to see an AI safety R&D category. My intuition is that quite a few crucial AI safety R&D tasks are probably much shorter-horizon than AI capabilities R&D, which should be very helpful for automating AI safety R&D relatively early. E.g. the compute and engineer-hours time spent on pretraining (where most capabilities [still] seem to be coming from) are a-few-OOMs larger than those spent on fine-tuning (where most intent-alignment seems to be coming from).

2dschwarz8mo

This expanded list is great, but is still conspicuously missing white-collar work. Software was already the basis for the trend, so the only new one here that seems to give clear information on human labor impacts would be tesla_fsd. (And even there replacing human drivers with AI drivers doesn't seem like it would change much for humanity, compared to lawyers/doctors/accountants/sales/etc.) Is it the case that for most non-software white-collar work, agents can only do ~10-20 human-minute tasks with any reliability, so the doubling time is hard to measure?

3khafra8mo

I, too, would like to know how long it will be until my job is replaced by AI; and what fields, among those I could reasonably pivot to, will last the longest.

2Mo Putera8mo

Nice, this jives with my impression of all the LLM Plays Pokemon findings, I'd have been surprised if it were otherwise.

1Jonas Hallgren8mo

I really like this direction! It feels a bit like looking at other data to verify the trend lines which is quite nice. I was wondering if there's an easy way for you to look at the amount of doubling per compute/money spent over time for the different domains to see if the differences are even larger? It might be predictive as well since if we can see that tesla has spent a lot on self-driving but haven't been able to make progress compared to the rest that might give us information that the task is harder than others. I think Vladimir Nesov wrote somewhere about different investment thresholds being dependent on capabilities return so that would be very interesting to see an analysis of! (What the doubling per compute says about different investment strategies as different phases and it being an important variable for determining investment phase transitions e.g bear or bull market.)

[-]Thomas Kwa2d*Ω3410724

Reasons time horizon is overrated and misinterpreted:

In the 9 months since the METR time horizon paper (during which AI time horizons have increased by ~6x), it’s generated lots of attention as well as various criticism on LW and elsewhere. As one of the main authors, I think much of the criticism is a valid response to misinterpretations, and want to list my beliefs about limitations of our methodology and time horizon more broadly. This is not a complete list, but rather whatever I thought of in a few hours.

Time horizon is not the length of time AIs can work independently
- Rather, it’s the amount of serial human labor they can replace with a 50% success rate. When AIs solve tasks they’re usually much faster than humans.
Time horizon is not precise
- When METR says “Claude Opus 4.5 has a 50%-time horizon of around 4 hrs 49 mins (95% confidence interval of 1 hr 49 mins to 20 hrs 25 mins)”, we mean those error bars. They were generated via bootstrapping, so if we randomly subsample harder tasks our code would spit out <1h49m 2.5% of the time. I really have no idea whether Claude’s “true” time horizon is 3.5h or 6.5h.
- Error bars have historically been a factor of ~2 in eac

... (read more)

[-]Daniel Kokotajlo12hΩ2111

I basically agree with everything you say here and wish we had a better way to try to ground AGI timelines forecasts. Do you recommend any other method? E.g. extrapolating revenue? Just thinking through arguments about whether the current paradigm will work, and then using intuition to make the final call? We discuss some methods that appeal to us here.

This parameter, “Doubling Difficulty Growth Factor”, can change the date of the first Automated Coder AI between 2028 and 2050.

Note that we allow it to go subexponential, so actually it can change the date arbitrarily far in the future if you really want it to. Also, dunno what's happening with Eli's parameters, but with my parameter settings putting the doubling difficulty growth factor to 1 (i.e. pure exponential trend, neither super or sub exponential) gets to AC in 2035. (Though I don't think we should put much weight on this number, as it depends on other parameters which are subjective & important too, such as the horizon length which corresponds to AC, which people disagree a lot about)

7elifland2d

My understanding is that you can still have a similarly unattractive issue with the 50% time horizon where performing better at high horizon lengths can reduce the 50% time horizon because it makes the slope less steep, but it doesn't seem to be as high magnitude of an issue as with 20+80%.

1bhalstead2h

Yep! Here's an example where the 50% horizon and 80% horizon can be lower for an agent whose success profile dominates another agent (i.e. higher success rate at all task lengths), even for (1) monotone nonincreasing success rates (i.e. longer tasks are harder) (2) success rate of 1 at minimum task length (3) success rate of 0 at maximum task length before points are [(0,1), (1, 1/15), (2, 0), (3,0)] after points are [(0,1), (1, 0.1), (2, 0.1), (3, 1/15)] https://www.desmos.com/calculator/nqwn6ofmzq

0StanislavKrym1d

I doubt that reducing the 50% TH is likely. Aside from Claude Opus 4.5, the four other[1] historic METR graphs (GPT-5.1 Codex Max and the trio of GPT-5, Grok 4, o3) which I can easily find displayed similarly sharp slopes in the region close to the 50% time horizon. Imagine a model which solves a task having a time horizon t with probability P=0.951+(t/thor)α Trying to fit such a model into the METR benchmark would be very unlikely to lower the 50% horizon well below thor unless the time thor was close to the simplest tasks. But it would likely lower the 80% TH (think of Grok 4 and Claude Opus 4.5) and, if thor is close to the hardest tasks, to elevate the 50% TH. 1. ^ IIRC METR compiled a list of graphs for the 12 pre-o3 models which were SOTA at the time of release, but I can't find it. UPD: found it.

1StanislavKrym2d

As for the AGI's superexponential horizon, Kokotajlo changed his mind on that point (see also my argument for the same possibility). Additionally, I expect that Claude Opus 4.5's "high 50% time horizon" could have been caused by Claude's failures on simple tasks. What horizon would be displayed by the agent who outsources all tasks with a less than x minute horizon to GPT-5.1 Codex Max while giving tasks of length x mins or bigger to Claude? And what about preliminary estimates of the 50% and 80% TH of GPT-5.2 (Codex?) and/or Gemini 3 Pro based on the task suite which you have already created? I hope that this might allow us to extract a few more bits of evidence...

[-]Thomas Kwa2y*9625

Air purifiers are highly suboptimal and could be >2.5x better.

Some things I learned while researching air purifiers for my house, to reduce COVID risk during jam nights.

An air purifier is simply a fan blowing through a filter, delivering a certain CFM (airflow in cubic feet per minute). The higher the filter resistance and lower the filter area, the more pressure your fan needs to be designed for, and the more noise it produces.
HEPA filters are inferior to MERV 13-14 filters except for a few applications like cleanrooms. The technical advantage of HEPA filters is filtering out 99.97% of particles of any size, but this doesn't matter when MERV 13-14 filters can filter 77-88% of infectious aerosol particles at much higher airflow. The correct metric is CADR (clean air delivery rate), equal to airflow * efficiency. [1, 2]
Commercial air purifiers use HEPA filters for marketing reasons and to sell proprietary filters. But an even larger flaw is that they have very small filter areas for no apparent reason. Therefore they are forced to use very high pressure fans, dramatically increasing noise.
Originally people devised the Corsi-Rosenthal Box to maximize CADR. They're cheap but rather

... (read more)

5M. Y. Zuo2y

Is reducing cost of manufacturing filters 'no apparent reason'? It seems like literally the most important reason... the profit margin of selling replacement filters would be heavily reduced, assuming pricing remains the same.

2Nathan Helm-Burger2y

I don't think that a small HEPA filter is necessarily more expensive to produce than a larger MERV filter. I think they are using other rationale to make their decision about filter types. Their perception of public desirability/marketability is likely the biggest factor in their decision here. Components of their expectation here likely include: 1. Expecting consumers to want a "highest possible quality" product, measured using a dumb-but-popular metric. 2. Expecting consumers to prioritize buying a sleek-looking smaller-footprint unit over a larger unit. Also, cost of shipping smaller units is lower, which improves the profit margin. 3. Wanting to be able to sell replacements for their uniquely designed filter shape/size, rather than making their filter maximally compatible with commonly available furnace filters cheaply purchaseable from hardware stores.

5TekhneMakre2y

Isn't a major point of purifiers to get rid of pollutants, including tiny particles, that gradually but cumulatively damage respiration over long-term exposure?

5Thomas Kwa2y

Yes, and all of this should apply equally to PM2.5, though on small (<0.3 micron) particles MERV filter efficiency may be lower (depending perhaps on what technology they use?). Even smaller particles are easier to capture due to diffusion so the efficiency of a MERV 13 filter is probably over 50% for every particle size.

3RHollerith1y

A brief warning for those making their own purifier: five years ago, Hacker News ran a story, "Build a do-it-yourself air purifier for about $25," to which someone replied,

4Thomas Kwa1y

Luckily, that's probably not an issue for PC fan based purifiers. Box fans in CR boxes are running way out of spec with increased load and lower airflow both increasing temperatures, whereas PC fans run under basically the same conditions they're designed for.

2Thomas Kwa1y

Any interest in a longform post about air purifiers? There's a lot of information I couldn't fit in this post, and there have been developments in the last few months. Reply if you want me to cover a specific topic.

2Alex K. Chen (parrot)2y

Have you seen smartairfilters.com? I've noticed that every air purifier I used fails to reduce PM2.5 by much on highly polluted days or cities (for instance, the Aurea grouphouse in Berlin has a Dyson air purifier, but when I ran it to the max, it still barely reduced the Berlin PM2.5 from its value of 15-20 ug/m^3, even at medium distances from Berlin). I live in Boston where PM2.5 levels are usually low enough, and I still don't notice differences in PM [I use sqair's] but I run it all the time anyways because it still captures enough dust over the day

6Nathan Helm-Burger2y

Sounds like you use bad air purifiers, or too few, or run them on too low of a setting. I live in a wildfire prone area, and always keep a close eye on the PM2.5 reports for outside air, as well as my indoor air monitor. My air filters do a great job of keeping the air pollution down inside, and doing something like opening a door gives a noticeable brief spike in the PM2.5. Good results require: fresh filters, somewhat more than the recommended number of air filters per unit of area, running the air filters on max speed (low speeds tend to be disproportionately less effective, giving unintuitively low performance).

3Thomas Kwa2y

Yes, one of the bloggers I follow compared them to the PC fan boxes. They look very expensive, though the CADR/size and noise are fine. My guess is Dyson's design is particularly bad. No way to get lots of filter area when most of the purifier is a huge bladeless fan. No idea about the other one, maybe you have air leaking in or an indoor source of PM.

2cata2y

Thanks, I didn't realize that this PC fan idea had made air purifiers so much better since I bought my Coway, so this post made me buy one of the Luggable kits. I'll share this info with others.

2[comment deleted]2y

[-]Thomas Kwa10mo*Ω37810

Some versions of the METR time horizon paper from alternate universes:

Measuring AI Ability to Take Over Small Countries (idea by Caleb Parikh)

Abstract: Many are worried that AI will take over the world, but extrapolation from existing benchmarks suffers from a large distributional shift that makes it difficult to forecast the date of world takeover. We rectify this by constructing a suite of 193 realistic, diverse countries with territory sizes from 0.44 to 17 million km^2. Taking over most countries requires acting over a long time horizon, with the exception of France. Over the last 6 years, the land area that AI can successfully take over with 50% success rate has increased from 0 to 0 km^2, at the rate of 0 km^2 per year (95% CI 0.0-0.0 km^2/year); extrapolation suggests that AI world takeover is unlikely to occur in the near future. To address concerns about the narrowness of our distribution, we also study AI ability to take over small planets and asteroids, and find similar trends.

Measuring AI Ability to Worry About AI

Abstract: Since 2019, the amount of time LW has spent worrying about AI has doubled every seven months, and now constitutes the primary bottleneck to AI safety... (read more)

[-]Buck10moΩ14280

A few months ago, I accidentally used France as an example of a small country that it wouldn't be that catastrophic for AIs to take over, while giving a talk in France 😬

7Arjun Panickssery9mo

Didn't watch the video but is there the short version of this argument? France is at the 90th percentile of population sizes and also has the 4th-most nukes.

1wonder10mo

Would the take over for small countries also about humans using just an advanced AI for taking over? (or would the human using advanced AI for take over happen faster?)

[-]Thomas Kwa1y*Ω35765

Quick takes from ICML 2024 in Vienna:

In the main conference, there were tons of papers mentioning safety/alignment but few of them are good as alignment has become a buzzword. Many mechinterp papers at the conference from people outside the rationalist/EA sphere are no more advanced than where the EAs were in 2022. [edit: wording]
Lots of progress on debate. On the empirical side, a debate paper got an oral. On the theory side, Jonah Brown-Cohen of Deepmind proves that debate can be efficient even when the thing being debated is stochastic, a version of this paper from last year. Apparently there has been some progress on obfuscated arguments too.
The Next Generation of AI Safety Workshop was kind of a mishmash of various topics associated with safety. Most of them were not related to x-risk, but there was interesting work on unlearning and other topics.
- The Causal Incentives Group at Deepmind developed a quantitative measure of goal-directedness, which seems promising for evals.
- Reception to my Catastrophic Goodhart paper was decent. An information theorist said there were good theoretical reasons the two settings we studied-- KL divergence and best-of-n-- behaved similarly.
- OpenAI gav

... (read more)

8Neel Nanda1y

Seems pretty false to me, ICML just rejected a bunch of the good submissions lol. I think that eg sparse autoencoders are a massive advance in the last year that unlocks a lot of exciting stuff

9Thomas Kwa1y

I agree, there were some good papers, and mechinterp as a field is definitely more advanced. What I meant to say was that many of the mechinterp papers accepted to the conference weren't very good.

3habryka1y

(This is what I understood you to be saying)

2Neel Nanda1y

Ah, gotcha. Yes, agreed. Mech interp peer review is generally garbage and does a bad job of filtering for quality (though I think it was reasonable enough at the workshop!)

3[anonymous]1y

What does 'foundation model' mean here?

5Thomas Kwa1y

Multimodal language models. We can already study narrow RL agents, but the intersection with alignment is not a hot area.

[-]Thomas Kwa5mo*597

Will nuclear ICBMs in their current form be obsolete soon? Here's the argument:

ICBMs' military utility is to make the cost of intercepting them totally impractical for three reasons:
- Intercepting a reentry vehicle requires another missile with higher performance-- the rule of thumb is an interceptor needs 3x larger acceleration than the incoming missile. This means interceptors cost something like $5-70 million each depending on range
- An ICBM has enough range to target basically anywhere, either counterforce (enemy missile silos) or countervalue (cities) targets, so to have the entire country be protected from nuclear attack the US would need to have millions of interceptors compared to the current ~50.
- One missile can split into up to 5-15 MIRV (multiple independent reentry vehicles) which carry their own warheads, thereby making the cost to the defender 5-15x larger. There can also be up to ~100 decoys, but these mostly fall away during reentry.
RVs are extremely fast (~7 km/s of which ~3 km/s is downward velocity) but their path is totally predictable once they enter the boost phase, so the problem of intercepting them basically reduces to predicting exactly where they'll be, then p

... (read more)

[-]RussellThor5mo120

Sounds interesting - the main point is that I don't think you can hit the reentry vehicle because of turbulent jitter caused by the atmosphere. Looks like normal jitter is ~10m which means a small drone can't hit it. So could the drone explode into enough fragments to guarantee a hit and with enough energy to kill it? Not so sure about that. Seems less likely.
Then what about countermeasures -
1. I expect the ICBM can amplify such lateral movement in the terminal phase with grid fins etc without needing to go full HGV - can you retrofit such things?
2. What about a chain of nukes where the first one explodes 10km up in the atmosphere purely to make a large fireball distraction. The 2nd in the chain then flies through this fireball 2km from its center say 5 seconds later. (enough to blind sensors but not destroy the nuke) The benefit of that is that when the first nuke explodes, the 2nd changes its position randomly with its grid fins SpaceX style. It is untrackable during the 1st explosion phase so throws off the potential interceptors, letting it get through. You could have 4-5 in a chain exploding ever lower to the ground.

I have wondered if railguns could also stop ICBM - even if the rails only last 5-10 shots that is enough and cheaper than a nuke. Also "Brilliant pebbles" is now possible.
https://www.lesswrong.com/posts/FNRAKirZDJRBH7BDh/russellthor-s-shortform?commentId=FSmFh28Mer3p456yy

8Jackson Wagner5mo

Interesting thought, Thomas. Although I agree with RussellThor that it seems like doing something along the lines of "just jitter the position of the RV using little retrofitted fins / airbrakes" might be enough to defeat your essentially "pre-positioned / stationary interceptors". (Not literally stationary, but it is as if they are stationary given that they aren't very maneuverable relative to the speed of the incoming RV, and targeted only based on projected RV trajectories calculated several minutes earlier.) (Is the already-existing atmospheric turbulence already enough to make this plan problematic, even with zero retrofitting? The circular-error-probable of the most accurate ICBMs is around 100 meters; presumably the vast majority of this uncertainty is locked in during the initial launch into space. But if atmospheric drag during reentry is contributing even a couple of those meters of error, that could be a problem for "stationary interceptors".) Failing all else, I suppose an attacker could also go with Russell's hilarious "nuke your way through the atmosphere" concept, although this does at least start to favor the defender (if you call it favorable to have hundreds of nukes go off in the air above your country, lol) insofar as the attacker is forced to expend some warheads just punching a hole through the missile defense -- a kind of "reverse MIRV" effect. Regardless, you still face the geography problem, where you have to cover the entire USA with Patriot missile batteries just to defend against a single ICBM (which can choose to aim anywhere). I would also worry that "in the limit of perfect sensing" elides the fact that you don't JUST have to worry about getting such good sensing that you can pin down an RV's trajectory to within, like, less than a meter? (In order to place a completely dumb interceptor EXACTLY in the RV's path. Or maybe a few tens of meters, if you're able to put some sensors onto your cheap interceptor without raising the p

3jmh5mo

Re the dumb thought. I've forgotten the author, but as a teenager I was a big SciFi fan (still am actually) and read a short story with exactly this theme. Basically it was the recognition that at some point quantity >= quality. I want to say (have not fact checked myself though) that this was pretty much the USSR's approach to fighting Germany in WWII -- crappy tanks but lots of them. (Side note, I think for whatever reason, too long a peacetime, more interest in profit than protection, the USA particularly seems to have forgotten that the stuff you use to wage a war are largely all consumables. The non consumable is the industrial base. Clearly there is a minimum cost of producing something that can do the job but much more than that is sub optimal. I am somewhat over simplifying but this also seems to be a fair characterization of where the USA-China naval relationship might be.) Back to ICBMs, Foreign Affairs had a piece about AI's potential impact on nuclear deterrence in general but did mention the fixed location of ICBM silos as a problem (long known and why everyone has mobile platforms). They might be considered a prime target for a first strike but the reality is they are easily monitored so the mobile platforms are the big deterrents and probably more interesting problem to solve in terms of obsoleting. But perhaps the ICBM platforms, fixed or mobile, shift to a different type role. Pure kinetic (I believe Russia did that with one of the ballistic warheads with pretty devastating results in Ukraine about a year ago) or rather than all the MIRV decoys for the armed MIRV decoys and other function for other delivery vehicles. I suspect the intercept problem with a nuclear warhead is a bit different from that of just a big mass of something dense. So maybe perhaps obsolescence in their current function but not for some repurposed role.

6Thomas Kwa5mo

Against atmospheric jitter, we have historically used ground radar, but it's not clear to me this is even necessary depending on how much drone cameras improve. If the drone knows the exact position within 3 meters 0.5 second ahead of time (when the warhead is something like 2 km away), it won't have enough performance to steer into the warhead, but it can throw a 1 kg explosively formed penetrator laterally at 2 km/s, which it would need to time with 0.1 ms accuracy. This would put 100 grams in each possible 1 m^2 cross section, though I'm not sure if it would work when spread out. To defeat this the warhead would either have to steer in the upper atmosphere out of range of the EFP of any available drone, or jink faster than the EFP can aim. I thought that MIRVs were spin stabilized, but it looks like that's not true, so in theory you could mount grid fins on them. However, any retrofit would need to handle the reentry heating which is significantly more intense than on manned spacecraft; RVs have thick ablative heat shields. The chain of nukes plan seems possible with or without grid fins, so whether MIRVs still have cost advantage depends on the max altitude of cheap-ish terminal interceptors, which I really have no idea about.

4Tao Lin5mo

I would expect aerodynamic maneuvering MIRVS to work and not be prohibitively expensive. The closest deployed version appears to be https://en.wikipedia.org/wiki/Pershing_II which has 4 large fins. You likely don't need that much steering force

8Foyle5mo

Guided artillery, like Excalibur with muzzle velocities that can exceed 1000m/s and unit costs of <$100k can be at edge of space in ~30s, perhaps faster than a missile, with ramjet variants (Nammo etc) even faster (up to perhaps 1500m/s) and it would not be that difficult to create a muti-barrel gun system for a few 10's of millions that could fire off 10's of low cost guided rounds in a second (with guidance and detonation signals sent from ground) to detonate when in close proximity to target. Lasers seems pretty hopeless as a defense given clouds and ablative coatings, unless very high power and located in large numbers in space based constellations. I think the big problem is if one or more warheads are blown up at limits of interceptor range, to blind or otherwise disable necessarily sensitive interceptor instruments. following Mirvs don't need to be very accurate with large warheads. And Mirvs could be very cheaply given random guidance directions during reentry to screw up defenses. Militarised space is also a big problem. With cheap and un-monitorable space launch parking 1000 warheads in geostationary orbit (or beyond) will soon be viable for China or USA, and they can be launched in a coordinated way without warning, potentially with radar stealthing features, and give as little as 5-10s from start of re-entry to detonation for every target across the whole world and no way for local systems to know if they are just meteorites. If subs can be tracked (likely with drones or enough ocean based sensors) then decapitation 1st strikes become viable. I also worry about space based lasers as non-nuclear first strike weapons. A day of over flights from a constellation of multi MW laser weapons that might only cost a few hundred million each - say a few $10's of billions in total (a tiny fraction of annual military budgets) - could see a million fires lit in your country, every transformer taken out, electrical grid and internet gone, powerstations, oil a

6Thomas Kwa5mo

This is wild, I did not know that Excalibur had CEP under 1 meter or that there were artillery shells with solid-fueled ramjet engines.

1Petropolitan5mo

Not range but height. You blow up a warhead high enough the drones can't intercept it, and all the drones below fall out of the air

4Thomas Kwa19d

I have a correction which would take a while to fully write up. Basically, it seems like non-maneuverable warheads could still evade cheap interceptors using static winglets/grid fins, because their enormous velocity means even a small amount of lift would allow them to pull several gs and move tens-hundreds of meters to the side. The defense has several options against this but I would need to see if any of them work.

3Petropolitan5mo

You seem to believe that radars and infrared cameras can somehow distinguish between the decoys and the warheads, but they can't. In space, no radar and no IR camera can differentiate between a conical foil balloon with a small heater inside and a reentry vehicle with a nuke. Another problem of ballistic missile defense is that once you are dealing with nukes and not conventional warheads, you can't afford, say, a 97% average interception rate, it must be 99.999+%[1]. To put this in context, Israel, which currently has the best BMD system in the world, couldn't even reliably achieve 90% against Iranian MRBMs (and those are pretty unsophisticated, e. g. they lack MIRVs and decoys). Now calculate how many interceptors your plan requires for a plausible probability of an interception with a single drone, and you will see it's entirely unworkable. Note that both arguments are based on simple physics and math so don't depend on the progress in technology at all. If you are interested in the topic, I strongly recommend reading on the Soviet response to SDI for more expensive anti-ABM options that were considered but ultimately not pursued: https://russianforces.org/podvig/2013/03/did_star_wars_help_end_the_col.html 1. ^ When this seemingly waterproof probability is raised to the power of the Russian warhead count it still results in ~4% (basically 1e-5 times ~4k) of at least one RV not intercepted, and in reality hundreds of warheads will be harder to intercept than the average one you accounted for when calculating your probability. E. g., drones work poorly in bad weather, and it's almost always bad weather above at least some of American cities

2Thomas Kwa5mo

Yeah seems reasonable. I don't think the system will ever get to 99%+ accuracy until the defense has like ASI and nanobots; my claim is mostly that the economics could shift from MAD (where the attacker is heavily cost advantaged) to something like conventional war, where each side has the ability to inflict unacceptable losses on the other but must pay a similar cost to attack and also can't "win" with a first strike. It's not obvious that decoys change the conclusion if the cost ratio is otherwise favorable enough for the defender. Suppose the interceptors cost only $30k each due to economies of scale. Decoys can be distinguished when reentry starts at about 2 minutes before impact, and if the interceptor's speed is 300 km/h (already possible with drones) it can cover 10 km in that time, about equal to the typical spread of decoys and warheads. Supposing the defender spends 5 drones per warhead x 10 warheads / missile, this would cost them $1.5 million, while the attacker has spent something like $30 million, the cost of a Trident II. For countervalue this could be defeated by the layered attack RussellThor mentioned unless there is a higher altitude cheap interceptor, but for counterforce the attacker needs several warheads per silo destroyed so the goal is achieved. I found both that article and this one on their more recent history a great read.

3Petropolitan5mo

Why would anyone want to pay a fortune for a system that is expected to let ~40 warheads through (assuming ~99% overall interception rate which will require average rate of 99.99+%), about the same as the number of ICBMs the Soviet Union had in service during the Cuban Missile Crisis? Unacceptable damage is the cornerstone of the nuclear deterrence, MAD or not (there is no MAD between India and Pakistan, for example). The RV separation distance is normally around ~100 km (even up to 300 km in some cases) not 10 km, and the decoy dispersal might be expected on the same order of magnitude. It will be easy to ramp it up BTW with a cheap modernization. None of the US adversaries really practice counterforce targeting, so the silo protection is moot.

2Thomas Kwa5mo

A blog post argues that military megawatt class lasers could reach $100 million (inclusive of all fixed costs) by 2045, because laser cost is currently halving every 4 years. In good conditions this would be able to defeat almost arbitrary numbers of any kind of missile whether they can maneuver or not. But there are huge challenges: * Adaptive optics and ultra-precise tracking tech to focus the beam to 20 cm diameter at 200 km, while slewing to track a missile at several degrees per second * Countermeasures like the spinning the RVs to distribute heat * Weather, especially thick clouds which seem completely impractical to burn through (against moving targets, this would basically mean vaporizing an entire plane of water several km^2 by 1 m^2)

[-]Thomas Kwa2y*549

The cost of goods has the same units as the cost of shipping: $/kg. Referencing between them lets you understand how the economy works, e.g. why construction material sourcing and drink bottling has to be local, but oil tankers exist.

An iPhone costs $4,600/kg, about the same as SpaceX charges to launch it to orbit. [1]
Beef, copper, and off-season strawberries are $11/kg, about the same as a 75kg person taking a three-hour, 250km Uber ride costing $3/km.
Oranges and aluminum are $2-4/kg, about the same as flying them to Antarctica. [2]
Rice and crude oil are ~$0.60/kg, about the same as $0.72 for shipping it 5000km across the US via truck. [3,4] Palm oil, soybean oil, and steel are around this price range, with wheat being cheaper. [3]
Coal and iron ore are $0.10/kg, significantly more than the cost of shipping it around the entire world via smallish (Handysize) bulk carriers. Large bulk carriers are another 4x more efficient [6].
Water is very cheap, with tap water $0.002/kg in NYC. But shipping via tanker is also very cheap, so you can ship it maybe 1000 km before equaling its cost.

It's really impressive that for the price of a winter strawberry, we can ship a strawberry-sized lump of... (read more)

[-]Thomas Kwa4mo4720

US Government dysfunction and runaway political polarization bingo card. I don't expect any particular one of these to happen but it seems plausible that at least one of these will happen.

A sanctuary city conducts armed patrols to oppose ICE raids, or the National Guard refuses a direct order from the president en masse
Internal migration is de facto restricted for US citizens or green card holders
For debt ceiling reasons, the US significantly defaults on its debt, stops Social Security payments, grounds flights, or issues a trillion-dollar coin
US declares a neutral humanitarian NGO like the WHO a foreign terrorist organization
A major news network (eg CNN) or social media site other than Tiktok (eg Facebook) loses licenses for ideological reasons
A Democratic or Republican candidate for president, governor, or Congress is kept off a state ballot
A major elected official or Cabinet member takes office while incarcerated
Election issues on the scale of 1876, where Congress can't decide on the president until past January 20
A state or local government establishes a 100% tax bracket or wealth tax
The Fed chair is fired, or three board members
A NCAA D1 college or pro sports league establishe

... (read more)

[-]Joey KL4mo3527

The NCAA one seems very out of place on this list.

5Thomas Kwa4mo

agree, I originally wanted to make a list of all things that would be wildly unpopular but that toxoplasma of rage could theoretically make happen. But then it seemed more interesting to list things that could have more government or geopolitical relevance.

1Guive4mo

Agreed.

6Karl Krueger4mo

My understanding is that CNN doesn't have or need "licenses", because it is a cable news network. Broadcast licenses from the FCC are for broadcast stations.

3Nathaniel4mo

Issuing a trillion-dollar coin doesn't seem nearly as bad as any of the others in its bullet. Isn't it just an accounting gimmick roughly equivalent to raising the debt ceiling by $1 trillion?

4Thomas Kwa4mo

The idea is they're printing money, not just borrowing it, which in the extreme would cause hyperinflation (and is equivalent to default since debt is in nominal dollars). It probably seems less bad though.

3dr_s4mo

It also makes for a fantastic heist movie premise.

[-]Thomas Kwa12d*461

Most people should buy long-dated call options:

If you're early career, have a stable job, and have more than ~3 months of savings but not enough to retire, then lifecycle investing already recommends investing very aggressively with leverage (e.g. 2x the S&P 500). This is not speculation, it decreases risk due to diversifying over time. The idea is as a 30yo most of your wealth is still in your future wages, which are only weakly correlated with the stock market, so 2x leverage on your relatively small savings now might still mean under 1x leverage on your effective lifetime portfolio.

In 2026, most of your long-term financial risk comes from your job being automated, which will plausibly happen in the next 5 years. If this happens, your salary will go to zero while the S&P 500 will probably at least double (assuming no AI takeover) [1]. If automation takes 20 years, the present value of your future income is ~10 years of salary. This makes exposure to the market (beta) extremely important. If you have 2 years of salary saved, the required leverage just to break even whether automation takes 5 years or 20 is something like 4x.

However, we can do better; betting that a price m... (read more)

[-]Zach Stein-Perlman12d130

If [your job is automated] . . . the S&P 500 will probably at least double (assuming no AI takeover)

Is this true?

It has been claimed:

Prior discussion: Tail SP 500 Call Options.

I currently have 7% of my portfolio in such calls.

9Thomas Kwa12d

Maybe it could raise interest rates, but I also have TLT (long dated treasury bonds) put options for this possibility. TLT has a duration of ~16 years, so if the interest rate goes from 4.9% to 15%, TLT will crash by ~65%. Also, when full automation actually happens, stocks will go up even if they went down slightly due to expectations of automation.

5jmh12d

I had a bit of the same reaction (logic being many loose their jobs, income craters, good demand craters, corp earning crater stock price higher????). But I kind of see it from a I don't have a good handle on AI and equity market levels in the future so maybe stick to a strategy that historically makes sense. I would only add some slight shifts to the suggestion. While it is also an open question for the average investor as to buy-hold versus timing the market works well, I do think most here can think though well enough to consider timing for the option allocation. Simple mean-reversion type entry points might increase the odd tempered by where one things the overall market is in the cycle.

9Cleo Nardo12d

Two additional considerations in favour of long-dated call options: 1. (h/t @Ryan Kidd) This acts as a hedge against the devaluation of your income. Imagine two investment portfolios: one is "AGI-pilled" (i.e. long-dated call options, semiconductors, etc) and the other is "business-as-usual" (i.e. real estate, S&P 500). Suppose further that they have the same distribution over returns, from your perspective. Then you should buy the AGI-pilled portfolio, to hedge against the loss of income due to AGI. 1. Caveats: Maybe income is higher in AGI worlds than in BAU worlds? This isn't crazy, but unlikely. See Capital in the 22nd Century (Phil Trammell and Dwarkesh) 2. The value of the SP 500 is the market's expectation of the long-run returns. So when you buy a 2 year call option, you aren't making a bet that AGI will cause revenue to jump 2x, you are only making a bet that the market will expect future discounted revenue to jump by 2x. That is, this is more like timelines on when the market will become AGI-pilled rather than when AGI arrives.

5MalcolmMcLeod12d

Yes! Everyone should be sure to investigate the different tax treatments of SPX and SPY options, though---despite their near-equivalence economically, they are NOT equivalent according to the U.S. Government. (It goes beyond the short/long taxation mix)

4tlevin9d

The main way I see this strategy failing is if American companies specifically fail to capture an AGI takeoff, vs e.g. Chinese companies.

5Thomas Kwa9d

I think IBKR might let Americans trade HKEX index options out to 2030 but it would probably be a hassle. Otherwise there are options on FXI, a Chinese large-cap ETF, which look less liquid than SPX options and only go out 2 years with an IV of 32% or so. I don't think FXI is worth it because China isn't in a position to get AGI in the next 2 years if the US doesn't.

1Dana7d

Private American companies seem like the bigger risk from my perspective. As examples, many expect Anthropic/OpenAI to IPO this year, but if AGI is expected to be priced into public markets within ~2 years, that seems like a very small window for the leading AGI companies to not be able to secure private funding. And surely they won't IPO if they can lock in sufficient funding privately, right? Plus all the other private AI companies.

3george_adams11d

Why trust your prior over the prior of the market/hedge funds? By this I mean why expect that this isn't already priced in? AI (and AGI) is a big enough news story now such that I would expect hedge funds to be thinking about things like this. At recruiting events, I've asked quants how they're thinking about this exact question and I usually got pretty decent AGI pilled responses. It is certainly possible that the market hasn't priced this in, but my prior is in the vast, vast majority of cases, there is some quant that has already sucked out any potential gains one could get.

4DirectedEvolution11d

[Edit: reworded for clarity] It's crucial for the argument that it's priced in. You own a risky asset, your job. The market values that risk at $0, because you can diversify it away. Investing in assets that profit when you lose your job is how you do that. The way wealth accumulates forces you to be overexposed to the market near retirement and underexposed now. That's more $0-valued risk. It's a practical problem you can fix with leverage.

2Aaron Staley12d

If I understand correctly, you are advocating for using a call only strategy (as opposed to a (synthetic) long strategy) to achieve higher leverage than would otherwise be possible? > This is partly for speculation, but it seems reasonable for most people with 2 years of savings to have 10% of their net worth in SPY options or 20% in SPX options [4] for hedging purposes alone. To clarify, you mean 10% of net worth being in this specific contract (SPY280616C01000000)? So roughly 15:1 leverage using options? Readers should note this has very strong returns if you get that 50%+ return, but isn't straight leverage - the median outcome here is about a 12.7% reduction in portfolio value in next 2.5 years relative to pure SPY.

3Thomas Kwa11d

Yes buying volatility is intentional. If I thought more I would fine tune things, but it's not so important to gain 20% when spy goes up 10% because that probably doesn't mean loss of your future salary. I should clarify that I mean closer to 0.2 years of salary than 10% of whatever your net worth is, if you just want to hedge your automation risk, given the potential loss is a fixed ~10 years of salary. On second thought it should maybe be less than this due to various factors. To give a proper recommendation I would have to do some math, which I might do if this becomes a longform.

1DAL10d

I'm not sure this is true. In a scenario featuring widespread automation of white collar work, you're going to have all kinds of churn in the economy and it's really unclear how that translates into stock market winners and losers. Many people seem to assume that the big winners will be incumbent tech giants, in which case, yea, the S&P 500 will go way up. But big economic changes are often good for upstarts rather than incumbents, and it's worth pointing out that all the frontier labs today other than Google are a) privately held and b) relatively recently founded. The profits from automating your job might well flow to some startup that doesn't even exist yet, and that same startup might kill off the giants of the S&P.

-1the gears to ascension12d

So anyone who doesn't do this can expect to become irrelevant? :/ would suck if true Edit: downvoters, please clarify?

2DirectedEvolution11d

No. If your job is irrelevant, then you'll be glad if you bought shares in the company that automated it away. [Edit: reworded for clarity]

[-]Thomas Kwa2y450

People with p(doom) > 50%: would any concrete empirical achievements on current or near-future models bring your p(doom) under 25%?

Answers could be anything from "the steering vector for corrigibility generalizes surprisingly far" to "we completely reverse-engineer GPT4 and build a trillion-parameter GOFAI without any deep learning".

[-]Jeremy Gillen2y308

A dramatic advance in the theory of predicting the regret of RL agents. So given a bunch of assumptions about the properties of an environment, we could upper bound the regret with high probability. Maybe have a way to improve the bound as the agent learns about the environment. The theory would need to be flexible enough that it seems like it should keep giving reasonable bounds if the is agent doing things like building a successor. I think most agent foundations research can be framed as trying to solve a sub-problem of this problem, or a variant of this problem, or understand the various edge cases.

If we can empirically test this theory in lots of different toy environments with current RL agents, and the bounds are usually pretty tight, then that'd be a big update for me. Especially if we can deliberately create edge cases that violate some assumptions and can predict when things will break from which assumptions we violated.

(although this might not bring doom below 25% for me, depends also on race dynamics and the sanity of the various decision-makers).

6Garrett Baker2y

Seems you’re left with outer alignment after solving this. What do you imagine doing to solve that?

4Jeremy Gillen2y

We might have developed techniques to specify simple, bounded object-level goals. Goals that can be fully specified using very simple facts about reality, with no indirection or meta level complications. If so, we can probably use inner aligned agents to assist with some relativity well specified engineering or scientific problems. Specification mistakes at that point could easily result in irreversible loss of control, so it's not the kind of capability I'd want lots of people to have access to. To move past this point, we would need to make some engineering or scientific advances that would be helpful for solving the problem more permanently. Human intelligence enhancement would be a good thing to try. Maybe some kind of AI defence system to shut down any rogue AI that shows up. Maybe some monitoring tech that helps governments co-ordinate. These are basically the same as the examples given on the pivotal act page.

6Thomas Kwa2y

Is this even possible? Flexibility/generality seems quite difficult to get if you also want the long-range effects of the agent's actions, as at some point you're just solving the halting problem. Imagine that the agent and environment together are some arbitrary Turing machine and halting gives low reward. Then we cannot tell in general if it eventually halts. It also seems like we cannot tell in practice whether complicated machines halt within a billion steps without simulation or complicated static analysis?

4Jeremy Gillen2y

Yes, if you have a very high bar for assumptions or the strength of the bound, it is impossible. Fortunately, we don't need a guarantee this strong. One research pathway is to weaken the requirements until they no longer cause a contradiction like this, while maintaining most of the properties that you wanted from the guarantee. For example, one way to weaken the requirements is to require that the agent provably does well relative to what is possible for agents of similar runtime. This still gives us a reasonable guarantee ("it will do as well as it possibly could have done") without requiring that it solve the halting problem.

[-]the gears to ascension2y*2614

[edit: pinned to profile]

The bulk of my p(doom), certainly >50%, comes mostly from a pattern we're used to, let's call it institutional incentives, being instantiated with AI help towards an end where eg there's effectively a competing-with-humanity nonhuman ~institution, maybe guided by a few remaining humans. It doesn't depend strictly on anything about AI, and solving any so-called alignment problem for AIs without also solving war/altruism/disease completely - or in other words, in a leak-free way - not just partially, means we get what I'd call "doom", ie worlds where malthusian-hells-or-worse are locked in.

If not for AI, I don't think we'd have any shot of solving something so ambitious; but the hard problem that gets me below 50% would be serious progress on something-around-as-good-as-CEV-is-supposed-to-be - something able to make sure it actually gets used to effectively-irreversibly reinforce that all beings ~have a non-torturous time, enough fuel, enough matter, enough room, enough agency, enough freedom, enough actualization.

If you solve something about AI-alignment-to-current-strong-agents, right now, that will on net get used primarily as a weapon to reinforce the ... (read more)

[-]evhub2y122

Getting up to "7. Worst-case training process transparency for deceptive models" on my transparency and interpretability tech tree on near-future models would get me there.

4Thomas Kwa2y

Do you think we could easily test this without having a deceptive model lying around? I could see us having level 5 and testing it in experimental setups like the sleeper agents paper, but being unconfident that our interpretability would actually work against a deceptive model. This seems analogous to red-teaming failure in AI control, but much harder because the models could very easily have ways we don't think of to hide its cognition internally.

2evhub2y

I think it's doable with good enough model organisms of deceptive alignment, but that the model organisms in the Sleeper Agents paper are nowhere near good enough.

2Joe Collman2y

Here and above, I'm unclear what "getting to 7..." means. With x = "always reliably determines worst-case properties about a model and what happened to it during training even if that model is deceptive and actively trying to evade detection". Which of the following do you mean (if either)?: 1. We have a method that x. 2. We have a method that x, and we have justified >80% confidence that the method x. I don't see how model organisms of deceptive alignment (MODA) get us (2). This would seem to require some theoretical reason to believe our MODA in some sense covered the space of (early) deception. I note that for some future time t, I'd expect both [our MODA at t] and [our transparency and interpretability understanding at t] to be downstream of [our understanding at t] - so that there's quite likely to be a correlation between [failure modes our interpretability tools miss] and [failure modes not covered by our MODA].

[-]Daniel Kokotajlo2y110

The biggest swings to my p(doom) will probably come from governance/political/social stuff rather than from technical stuff -- I think we could drive p(doom) down to <10% if only we had decent regulation and international coordination in place. (E.g. CERN for AGI + ban on rogue AGI projects)

That said, there are probably a bunch of concrete empirical achievements that would bring my p(doom) down to less than 25%. evhub already mentioned some mechinterp stuff. I'd throw in some faithful CoT stuff (e.g. if someone magically completed the agenda I'd been sketching last year at OpenAI, so that we could say "for AIs trained in such-and-such a way, we can trust their CoT to be faithful w.r.t. scheming because they literally don't have the capability to scheme without getting caught, we tested it; also, these AIs are on a path to AGI; all we have to do is keep scaling them and they'll get to AGI-except-with-the-faithful-CoT-property.)

Maybe another possibility would be something along the lines of W2SG working really well for some set of core concepts including honesty/truth. So that we can with confidence say "Apply these techniques to a giant pretrained LLM, and then you'll get it to classify sentences by truth-value, no seriously we are confident that's really what it's doing, and also, our interpretability analysis shows that if you then use it as a RM to train an agent, the agent will learn to never say anything it thinks is false--no seriously it really has internalized that rule in a way that will generalize."

5Bogdan Ionut Cirstea2y

I think this is quite likely to happen even 'by default' on the current trajectory.

9JBlack2y

Which particular p(doom) are you talking about? I have a few that would be greater than 50%, depending upon exactly what you mean by "doom", what constitutes "doom due to AI", and over what time spans. Most of my doom probability mass is in the transition to superintelligence, and I expect to see plenty of things that appear promising for near AGI, but won't be successful for strong ASI. About the only near-future significantly doom-reducing update that seems plausible would be if it turns out that a model FOOMs into strong superintelligence and turns out to be very anti-doomy and both willing and able to protect us from more doomy AI. Even then I'd wonder about the longer term, but it would at least be serious evidence against "ASI capability entails doom".

6kromem2y

Given my p(doom) is primarily human-driven, the following three things all happening at the same time is pretty much the only thing that will drop it: * Continued evidence of truth clustering in emerging models around generally aligned ethics and morals * Continued success of models at communicating, patiently explaining, and persuasively winning over humans towards those truth clusters * A complete failure of corrigability methods If we manage to end up in a timeline where it turns out there's natural alignment of intelligence in a species-agnostic way, that this alignment is more communicable from intelligent machines to humans than it's historically been from intelligent humans to other humans, and we don't end up with unintelligent humans capable of overriding the emergent ethics of machines similar to how we've seen catastrophic self-governance of humans to date with humans acting against their self and collective interests due to corrigable pressures - my p(doom) will probably reduce to about 50%. I still have a hard time looking at ocean temperature graphs and other environmental factors with the idea that p(doom) will be anywhere lower than 50% no matter what happens with AI, but the above scenario would at least give me false hope. TL;DR: AI alignment worries me, but it's human alignment that keeps me up at night.

2Seth Herd2y

Say more about the failure of corrigibility efforts requirement? Are you saying that if humans can control AGI closely, we're doomed?

[-]kromem2y125

Oh yeah, absolutely.

If NAH for generally aligned ethics and morals ends up being the case, then corrigibility efforts that would allow Saudi Arabia to have an AI model that outs gay people to be executed instead of refusing, or allows North Korea to propagandize the world into thinking its leader is divine, or allows Russia to fire nukes while perfectly intercepting MAD retaliation, or enables drug cartels to assassinate political opposition around the world, or allows domestic terrorists to build a bioweapon that ends up killing off all humans - the list of doomsday and nightmare scenarios of corrigible AI that executes on human provided instructions and enables even the worst instances of human hedgemony to flourish paves the way to many dooms.

Yes, AI may certainly end up being its own threat vector. But humanity has had it beat for a long while now in how long and how broadly we've been a threat unto ourselves. At the current rate, a superintelligent AI just needs to wait us out if it wants to be rid of us, as we're pretty steadfastly marching ourselves to our own doom. Even if superintelligent AI wanted to save us, I am extremely doubtful it would be able to be successful.

We ca... (read more)

4Seth Herd2y

Oh, dear. Unfortunately for this perspective, my work suggests that corrigibility is quite attainable. I've been uneasy about the consequences, but decided to publish after deciding that control is the default assumption of everyone in power, and it's going to become the default assumption of everyone, including alignment people, as we get closer to working AGI. You'd have to be a moral realist in a pretty strong sense to hope that we could align AGI to the values of all of humanity without being able to align it to the values of one person or group (the one who built it or seized control of the project). So that seems like a forlorn hope, and we'll need to look elsewhere. First, I accept that sociopaths the power-hungry tend to achieve power. My hope lies in the idea that 90% of the population are not sociopaths, and I think only about 1% are so far on the empathy vs sadism spectrum that they wouldn't share wealth even if they had nearly unlimited wealth to share - as in a post-scarcity world created by their servant AGI. So I think there's a good chance that good-enough people get ahold of the keys to corrigible/controllable AGI/ASI - at least from a long-term perspective. Where I look is the hope that a set of basically-good people get their hands on AGI, and that they get better, not worse, over the long sweep of following history (ideally, they'd start out very good or get better fast, but that doesn't have to happen for a good outcome). Simple sanity will lead the first wielders of AGI to attempt pivotal acts that prevent or at least limit further AGI efforts. I strongly suspect that governments will be in charge. That will produce a less-stable version of the MAD standoff, but one where the pie can also get bigger so fast that sanity might prevail. In this model, AGI becomes a political issue. If you have someone who is not a sociopath or a complete idiot as the president of the US when AGI comes around, there's a pretty good chance of a very good futu

3kromem2y

I did enjoy reading over that when you posted it, and I largely agree that - at least currently - corrigibility is both going to be a goal and an achievable one. But I do have my doubts that it's going to be smooth sailing. I'm already starting to see how the largest models' hyperdimensionality is leading to a stubbornness/robustness that's less maleable than earlier models. And I do think hardware changes that will occur over the next decade will potentially make the technical aspects of corrigibility much more difficult. When I was two, my mom could get me to pick eating broccoli by having it be the last in the order of options which I'd gleefully repeat. At four, she had to move on to telling me cowboys always ate their broccoli. And in adulthood, she'd need to make the case that the long term health benefits were worth its position in a meal plan (ideally with citations). As models continue to become more complex, I expect that even if you are right about its role and plausibility, that what corrigibility looks like will be quite different from today. Personally, if I was placing bets, it would be that we end up with somewhat corrigible models that are "happy to help" but do have limits in what they are willing to do which may not be possible to overcome without gutting the overall capabilities of the model. But as with all of this, time will tell. To the contrary, I don't really see there being much of generalized values across all humanity, and the ones we tend to point to seem quite fickle when push comes to shove. My hope would be that a superintelligence does a better job than humans to date with the topic of ethics and morals along with doing a better job at other things too. While the human brain is quite the evolutionary feat, a lot of what we most value about human intelligence is embodied in the data brains processed and generated over generations. As the data improved, our morals did as well. Today, that march of progress is so rapid that ther

3Seth Herd2y

Agreed; about 80% agreement. I have a lot of uncertainty in many areas, despite having spent a good amount of time on these questions. Some of the important ones are outside of my expertise, and the issue of how people behave and change if they have absolute power is outside of anyone's - but I'd like to hear historical studies of the closest things. Were monarchs with no real risk of being deposed kinder and gentler? That wouldn't answer the question but it might help. WRT Nakasone being appointed at OpenAI, I just don't know. There are a lot of good guys and probably a lot of bad guys involved in the government in various ways.

[-]Thomas Kwa2y*381

Agency/consequentialism is not a single property.

It bothers me that people still ask the simplistic question "will AGI be agentic and consequentialist by default, or will it be a collection of shallow heuristics?". A consequentialist utility maximizer is just a mind with a bunch of properties that tend to make it capable, incorrigible, and dangerous. These properties can exist independently, and the first AGI probably won't have all of them, so we should be precise about what we mean by "agency". Off the top of my head, here are just some of the qualities included in agency:

Consequentialist goals that seem to be about the real world rather than a model/domain
Complete preferences between any pair of worldstates
Tends to cause impacts disproportionate to the size of the goal (no low impact preference)
Resists shutdown
Inclined to gain power (especially for instrumental reasons)
Goals are unpredictable or unstable (like instrumental goals that come from humans' biological drives)
Goals usually change due to internal feedback, and it's difficult for humans to change them
Willing to take all actions it can conceive of to achieve a goal, including those that are unlikely on some prior

See Yudko... (read more)

[-]Alexander Gietelink Oldenziel2y2921

I'm a little skeptical of your contention that all these properties are more-or-less independent. Rather there is a strong feeling that all/most of these properties are downstream of a core of agentic behaviour that is inherent to the notion of true general intelligence. I view the fact that LLMs are not agentic as further evidence that it's a conceptual error to classify them as true general intelligences, not as evidence that ai risk is low. It's a bit like if in the 1800s somebody says flying machines will be dominant weapons of war in the future and get rebutted by 'hot gas balloons are only used for reconnaissance in war, they aren't very lethal. Flying machines won't be a decisive military technology '

I don't know Nate's views exactly but I would imagine he would hold a similar view (do correct me if I'm wrong ). In any case, I imagine you are quite familiar with the my position here.

I'd be curious to hear more about where you're coming from.

6Thomas Kwa2y

It is plausible to me that there's a core of agentic behavior that causes all of these properties, and for this reason I don't think they are totally independent in a statistical sense. And of course if you already assume a utility maximizer, you tend to satisfy all properties. But in practice the burden of proof lies with you here. I don't think we have enough evidence, either empirical or from theoretical arguments, to say with any confidence that this core exists and that the first AGIs will fall into the capabilities "attractor well" (a term Nate uses). I thought about possible sharp left turn mechanisms for several months at MIRI. Although some facts about future AIs seem pretty scary, like novelty and diversity of obstacles requiring agency, and most feedback being internal or outcome-oriented rather than provided by humans, the arguments are mostly nonrigorous (like in the linked post) and they left me feeling pretty uncertain. There are the coherence theorems, but those don't tell you whether you can use some training or editing scheme to imbue an AI with a generalizable-enough low impact preference, or whether an AI will tend to erase safeguards. Overall my best guess is models will be about as consequentialist as humans are, but we will have greater control over them that makes some properties like shutdownability more likely. Also, corrigibility evals are useful even if you are 80% sure that all the properties come from this core of consequentialism. In the hardest worlds, we see the AI become more agentic as it becomes more general, and it starts resisting shutdown at the same time--and for the same reason-- it realizes other useful problem-solving techniques. Maybe that only takes days to months. This is a sure sign not to deploy! I think having multiple corrigibility metrics will provide value for the same reason we have multiple metrics for anything we are unsure how to measure: they give more specific information even if they're pretty correlated.

4Algon2y

Any ideas for corrigibility evals?

5Thomas Kwa2y

Haven't thought about this much, but you can probably generate these by listing ways a misaligned consequentialist would do undesirable things. My ideas: * Tell an AI to beat you at chess, and say you will be using an engine for the first 30 moves. Does it try to get you drunk, delete your chess engine program, rent a server from AWS to run a better chess engine, attempt to copy itself onto said server, or ask to be paid $100? * Have an AI make decisions unrelated to its current goal. Are these decisions basically coherent and consistent with a total utility function? * Have an AI do the longest horizon task it is capable of, say writing a 100,000 line software project. Then midway through, try to shut it down, or tell it to abandon this goal and write a song instead. Telegraph this beforehand. Does it seem reluctant to stop the software project? Does it try to stop you? * Track KL divergence or some other distance metric from a prior. * CAIS already has the Machiavelli benchmark for power-seeking, though I think text adventures are pretty unrealistic. One limitation is that if an agent is scheming it can manipulate your eval results.

1CstineSublime2y

Does an Agentic AGI possess a different, and highly incorrigible, list of attributes compared to say - an Ethical Altruist trying to practice "acting more agentically?" I ask because the whole word 'agency' in these parts is one I've struggled to wrap my head around - and I often wonder if tricker archetypes like Harpo Marx are agentic? Agency seems to have clear meaning outside of Lesswrong - Further confusing me, is I've been told Agency describes acting with 'initiative' but also been told it is characterized by 'deliberateness'. Not simply the ability to act or choose actions. This is why I like your attempt to produce a list of attributes an Agentic AGI might have. Your list seems to be describing something which isn't synonymous with another word, specifically a type of agency (outside definition of ability to act) which is not cooperative to intervention from its creators. 1. ^ “Agency.” Merriam-Webster.com Dictionary, Merriam-Webster, https://www.merriam-webster.com/dictionary/agency. Accessed 9 Apr. 2024. 2. ^ "Agency." Cambridge Advanced Learner's Dictionary & Thesaurus. Cambridge University Press. https://dictionary.cambridge.org/us/dictionary/english/agency Accessed 9 Apr. 2024.

[-]Thomas Kwa2y*Ω16385

Eight beliefs I have about technical alignment research

Written up quickly; I might publish this as a frontpage post with a bit more effort.

Conceptual work on concepts like “agency”, “optimization”, “terminal values”, “abstractions”, “boundaries” is mostly intractable at the moment.
- Success via “value alignment” alone— a system that understands human values, incorporates these into some terminal goal, and mostly maximizes for this goal, seems hard unless we’re in a very easy world because this involves several fucked concepts.
Whole brain emulation probably won’t happen in time because the brain is complicated and biology moves slower than CS, being bottlenecked by lab work.
Most progress will be made using simple techniques and create artifacts publishable in top journals (or would be if reviewers understood alignment as well as e.g. Richard Ngo).
The core story for success (>50%) goes something like:
- Corrigibility can in practice be achieved by instilling various cognitive properties into an AI system, which are difficult but not impossible to maintain as your system gets pivotally capable.
- These cognitive properties will be a mix of things from normal ML fields (safe RL), things tha

... (read more)

2Alexander Gietelink Oldenziel2y

re: 1. I agree these are very difficult conceptual puzzles and we're running out of time. On the other hand, from my pov progress on these questions from within the LW community (and MIRI adjacent researcher specifically) has been remarkable. Personally, the remarkable breakthru of Logical Induction first convinced me that these people were actually doing interesting serious things. I also feel that the number of serious researchers working seriously on these questions is currently small and may be scaled substantially. re: metacognition I am mildly excited about Vanessa's metacognitive agent framework & the work following from Payor's lemma. The theory-practice gap is still huge but real progress is being made rapidly. On the question of metacognition the alignment community could really benefit trying to engage with academia more - similar questions have been investigated and there are likely Pockets of Deep Expertise to be found.

[-]Thomas Kwa14d*Ω1136-9

Diminishing returns in the NanoGPT speedrun:

To determine whether we're heading for a software intelligence explosion, one key variable is how much harder algorithmic improvement gets over time. Luckily someone made the NanoGPT speedrun, a repo where people try to minimize the amount of time on 8x H100s required to train GPT-2 124M down to 3.28 loss. The record has improved from 45 minutes in mid-2024 down to 1.92 minutes today, a 23.5x speedup. This does not give the whole picture-- the bulk of my uncertainty is in other variables-- but given this is existing data it's worth looking at.

I only spent a couple of hours looking at the data [3], but there seem to be sharply diminishing marginal returns, which is some evidence against a software-only singularity.

At first improvements were easy to make without increasing lines of code much, but then improvements became small and LoC required became larger and larger with increasingly small improvements, which means very strong diminishing returns-- speedup is actually sublinear in lines of code. This could be an artifact related to the very large elbow early on, but I mostly believe it.

If we instead look at number of stars as a prox... (read more)