Air purifiers are highly suboptimal and could be >2.5x better.

Some things I learned while researching air purifiers for my house, to reduce COVID risk during jam nights.

• An air purifier is simply a fan blowing through a filter, delivering a certain CFM (airflow in cubic feet per minute). The higher the filter resistance and lower the filter area, the more pressure your fan needs to be designed for, and the more noise it produces.
• HEPA filters are inferior to MERV 13-14 filters except for a few applications like cleanrooms. The technical advantage of HEPA filters is filtering out 99.97% of particles of any size, but this doesn't matter when MERV 13-14 filters can filter 77-88% of infectious aerosol particles at much higher airflow. The correct metric is CADR (clean air delivery rate), equal to airflow * efficiency. [1, 2]
• Commercial air purifiers use HEPA filters for marketing reasons and to sell proprietary filters. But an even larger flaw is that they have very small filter areas for no apparent reason. Therefore they are forced to use very high pressure fans, dramatically increasing noise.
• Originally people devised the Corsi-Rosenthal Box to maximize CADR. They're cheap but rather

Will nuclear ICBMs in their current form be obsolete soon? Here's the argument:

• ICBMs' military utility is to make the cost of intercepting them totally impractical for three reasons:
  • Intercepting a reentry vehicle requires another missile with higher performance-- the rule of thumb is an interceptor needs 3x larger acceleration than the incoming missile. This means interceptors cost something like $5-70 million each depending on range
  • An ICBM has enough range to target basically anywhere, either counterforce (enemy missile silos) or countervalue (cities) targets, so to have the entire country be protected from nuclear attack the US would need to have millions of interceptors compared to the current ~50.
  • One missile can split into up to 5-15 MIRV (multiple independent reentry vehicles) which carry their own warheads, thereby making the cost to the defender 5-15x larger. There can also be up to ~100 decoys, but these mostly fall away during reentry.
• RVs are extremely fast (~7 km/s of which ~3 km/s is downward velocity) but their path is totally predictable once they enter the boost phase, so the problem of intercepting them basically reduces to predicting exactly where they'll be, then p

US Government dysfunction and runaway political polarization bingo card. I don't expect any particular one of these to happen but it seems plausible that at least one of these will happen.

• A sanctuary city conducts armed patrols to oppose ICE raids, or the National Guard refuses a direct order from the president en masse
• Internal migration is de facto restricted for US citizens or green card holders
• For debt ceiling reasons, the US significantly defaults on its debt, stops Social Security payments, grounds flights, or issues a trillion-dollar coin
• US declares a neutral humanitarian NGO like the WHO a foreign terrorist organization
• A major news network (eg CNN) or social media site other than Tiktok (eg Facebook) loses licenses for ideological reasons
• A Democratic or Republican candidate for president, governor, or Congress is kept off a state ballot
• A major elected official or Cabinet member takes office while incarcerated
• Election issues on the scale of 1876, where Congress can't decide on the president until past January 20
• A state or local government establishes a 100% tax bracket or wealth tax
• The Fed chair is fired, or three board members
• A NCAA D1 college or pro sports league establishe

People with p(doom) > 50%: would any concrete empirical achievements on current or near-future models bring your p(doom) under 25%?

Answers could be anything from "the steering vector for corrigibility generalizes surprisingly far" to "we completely reverse-engineer GPT4 and build a trillion-parameter GOFAI without any deep learning".

Agency/consequentialism is not a single property.

It bothers me that people still ask the simplistic question "will AGI be agentic and consequentialist by default, or will it be a collection of shallow heuristics?". A consequentialist utility maximizer is just a mind with a bunch of properties that tend to make it capable, incorrigible, and dangerous. These properties can exist independently, and the first AGI probably won't have all of them, so we should be precise about what we mean by "agency". Off the top of my head, here are just some of the qualities included in agency:

• Consequentialist goals that seem to be about the real world rather than a model/domain
• Complete preferences between any pair of worldstates
• Tends to cause impacts disproportionate to the size of the goal (no low impact preference)
• Resists shutdown
• Inclined to gain power (especially for instrumental reasons)
• Goals are unpredictable or unstable (like instrumental goals that come from humans' biological drives)
• Goals usually change due to internal feedback, and it's difficult for humans to change them
• Willing to take all actions it can conceive of to achieve a goal, including those that are unlikely on some prior

See Yudko... (read more)

I think the framing "alignment research is preparadigmatic" might be heavily misunderstood. The term "preparadigmatic" of course comes from Thomas Kuhn's The Structure of Scientific Revolutions. My reading of this says that a paradigm is basically an approach to solving problems which has been proven to work, and that the correct goal of preparadigmatic research should be to do research generally recognized as impressive.

For example, Kuhn says in chapter 2 that "Paradigms gain their status because they are more successful than their competitors in solving a few problems that the group of practitioners has come to recognize as acute." That is, lots of researchers have different ontologies/approaches, and paradigms are the approaches that solve problems that everyone, including people with different approaches, agrees to be important. This suggests that to the extent alignment is still preparadigmatic, we should try to solve problems recognized as important by, say, people in each of the five clusters of alignment researchers (e.g. Nate Soares, Dan Hendrycks, Paul Christiano, Jan Leike, David Bau). 

I think this gets twisted in some popular writings on LessWrong. John Wentworth w... (read more)

In a world with an AI pause, good management is much more important than the pause length. If the pause is government-mandated and happens late, I would prefer a 1-year pause with say Buck Shlegeris in charge over a 3-year pause with the average AI lab lead in charge.

The reason is that most of the safety research during the pause will be done by AIs. In a pause scenario where >50% of resources are temporarily dedicated to safety and >10x uplift is technically plausible, the speed of research is mainly limited by the level of AI we trust and our ability to understand the AI's findings. The world probably can't pause indefinitely, and it may not be desirable to do so. So what is the basic strategy for a finite AI pause?

• Use superhuman AIs as that are smart as possible to do safety research, but not so capable that they're misaligned. This may require avoiding use of latest gen models, or advancing capabilities and using the next gen only for safety.
• Use an elaborate monitoring framework and incrementally apply the safety research generated to increase the maximum capability level we can run at without being sabotaged when models are misaligned.
• Prevent capability advances from bei

If you disagree with much of IABIED but are still worried about AI risk, maybe the question to ask is "will the radical flank effect be positive or negative on mainstream AI safety movements?", which seems more useful than "do I on net agree or disagree?" or "will people taking this book at face value do useful or anti-useful things?" Here's what Wikipedia has to say on the sign of a radical flank effect:

It's difficult to tell without hindsight whether the radical flank of a movement will have positive or negative effects.^[2] However, following are some factors that have been proposed as making positive effects more likely:

• Greater differentiation between moderates and radicals in the presence of a weak government.^{[2][13][14]: 411}As Charles Dobson puts it: "To secure their place, the new moderates have to denounce the actions of their extremist counterparts as irresponsible, immoral, and counterproductive. The most astute will quietly encourage 'responsible extremism' at the same time."^[15]
• Existing momentum behind the cause. If change seems likely to happen anyway, then governments are more willing to accept moderate reforms in order to quell radicals.^[2]
• Radicalism during the peak

Possible post on suspicious multidimensional pessimism:

I think MIRI people (specifically Soares and Yudkowsky but probably others too) are more pessimistic than the alignment community average on several different dimensions, both technical and non-technical: morality, civilizational response, takeoff speeds, probability of easy alignment schemes working, and our ability to usefully expand the field of alignment. Some of this is implied by technical models, and MIRI is not more pessimistic in every possible dimension, but it's still awfully suspicious.

I strongly suspect that one of the following is true:

• the MIRI "optimism dial" is set too low
• everyone else's "optimism dial" is set too high. (Yudkowsky has said this multiple times in different contexts)
• There are common generators that I don't know about that are not just an "optimism dial", beyond MIRI's models

I'm only going to actually write this up if there is demand; the full post will have citations which are kind of annoying to find.

Maybe this is too tired a point, but AI safety really needs exercises-- tasks that are interesting, self-contained (not depending on 50 hours of readings), take about 2 hours, have clean solutions, and give people the feel of alignment research.

I found some of the SERI MATS application questions better than Richard Ngo's exercises for this purpose, but there still seems to be significant room for improvement. There is currently nothing smaller than ELK (which takes closer to 50 hours to develop a proposal for and properly think about it) that I can point technically minded people to and feel confident that they'll both be engaged and learn something.

The uplift equation:

What is required for AI to provide net speedup to a software engineering project, when humans write higher quality code than AIs? It depends how it's used.

Cursor regime

In this regime, similar to how I use Cursor agent mode, the human has to read every line of code the AI generates, so we can write:

$\text{Speedup factor}=\frac{t_H}{t_{AI+H}}=\frac{t_H}{t_{prompt}+t_{AIgen}+t_{check}+p_{fail}\cdot t_H}$

Where

• $t_H$ is the time for the human to write the code, either from scratch or after rejecting an AI suggestion
• $t_{AIgen}$ is the time for the AI to generate the code in tokens per second.
• $t_{check}$ is the time for the human to check the code, accept or reject it, and make any minor revisions, in order to bring the code quality and probability of bugs equal with human-written code ($\Delta p_{bug}=0$)
• $p_{fail}$ is the fraction of AI suggestions that are rejected entirely.

Note this neglects other factors like code review time, code quality, bugs that aren't caught by the human, or enabling things the human can't do.

Autonomous regime

In this regime the AI is reliable enough that the human doesn't check all the code for bugs, and instead eats the chance of costly bugs entering the codebase.

$\text{Speedup factor}=\frac{t_H}{t_{AI}}=\frac{t_H}{t_{prompt}+t_{AIgen}}$... (read more)

GDM paper: Evaluating the Goal-Directedness of Large Language Models

Tom Everitt, Rohin Shah, and others from GDM attempt to measure "whether LLMs use their capabilities towards their given goal". Unlike previous work, their measure is not just rescaled task performance-- rather, an AI is goal-directed if it uses its capabilities effectively. A model that is not goal-directed when attempting a task will have capabilities but not properly use them. Thus, we can measure goal-directedness by comparing a model's actual performance to how it should perform if it were using its full capabilities on subtasks.

This kind of measure could in principle be useful to measure:

1. prerequisites of scheming: maybe a model is not capable of scheming until it reaches some level of goal-directedness, and we can forecast when that happens
2. elicitation: a normally goal-directed model that is not trying on evals may be underelicited
3. sandbagging: if an AI unexpectedly has low goal-directedness on evals, this could be a warning sign for sandbagging.

Unfortunately, it's mostly saturated already-- Gemini 2.0 and Claude 3.7 are over 95%. Even GPT-4 gets over 70%.

So until we measure models on tasks where they are curre... (read more)

Will we ever have Poké Balls in real life? How fast could they be at storing and retrieving animals? Requirements:

• Made of atoms, no teleportation or fantasy physics.
• Small enough to be easily thrown, say under 5 inches diameter
• Must be able to disassemble and reconstruct an animal as large as an elephant in a reasonable amount of time, say 5 minutes, and store its pattern digitally
• Must reconstruct the animal to enough fidelity that its memories are intact and it's physically identical for most purposes, though maybe not quite to the cellular level
• No external power source
• Works basically wherever you throw it, though it might be slower to print the animal if it only has air to use as feedstock mass or can't spread out to dissipate heat
• Should not destroy nearby buildings when used
• Animals must feel no pain during the process

It feels pretty likely to me that we'll be able to print complex animals eventually using nanotech/biotech, but the speed requirements here might be pushing the limits of what's possible. In particular heat dissipation seems like a huge challenge; assuming that 0.2 kcal/g of waste heat is created while assembling the elephant, which is well below what elephants need t... (read more)

Tech tree for worst-case/HRAD alignment

Here's a diagram of what it would take to solve alignment in the hardest worlds, where something like MIRI's HRAD agenda is needed. I made this months ago with Thomas Larsen and never got around to posting it (mostly because under my worldview it's pretty unlikely that we can, or need to, do this), and it probably won't become a longform at this point. I have not thought about this enough to be highly confident in anything.
 

• This flowchart is under the hypothesis that LLMs have some underlying, mysterious algorithms and data structures that confer intelligence, and that we can in theory apply these to agents constructed by hand, though this would be extremely tedious. Therefore, there are basically three phases: understanding what a HRAD agent would do in theory, reverse-engineering language models, and combining these two directions. The final agent will be a mix of hardcoded things and ML, depending on what is feasible to hardcode and how well we can train ML systems whose robustness and conformation to a spec we are highly confident in.
• Theory of abstractions: Also called multi-level models. A mathematical framework for a world-model tha

Superstable proteins: A team from Nanjing University just created a protein that's 5x stronger against unfolding than normal proteins and can withstand temperatures of 150C. The upshot from some analysis on X seems to be:

• They used 3 different AI tools (RFdiffusion, ProteinMPNN, ESMFold/AlphaFold2), plus molecular dynamics software
• The protein uses standard amino acids (edit: and was made in a standard bacterial cell)
• The strength comes from increasing the number of backbone hydrogen bonds from 4 to 33.

So why is this relevant? It's basically the first step towards nanotech. Because standard proteins aren't strong enough to manipulate molecular fragments, Drexler's original guess at the path to nanotech was: regular proteins assemble crosslinked proteins, which assemble hybrid nanotech, which assemble full nanotech. At each stage the energy scale increases, and the systems become increasingly capable.

It's plausible to me that within 15-30 years, enzymes like these superstable proteins (but much more advanced) will build stronger proteins which build hybrid systems which build molecular assemblers, until we have real-life Poke Balls that can print animals in 15 seconds.

I was going to write an April Fool's Day post in the style of "On the Impossibility of Supersized Machines", perhaps titled "On the Impossibility of Operating Supersized Machines", to poke fun at bad arguments that alignment is difficult. I didn't do this partly because I thought it would get downvotes. Maybe this reflects poorly on LW?

I'm looking for AI safety projects with people with some amount of experience. I have 3/4 of a CS degree from Caltech, one year at MIRI, and have finished the WMLB and ARENA bootcamps. I'm most excited about making activation engineering more rigorous, but willing to do anything that builds research and engineering skill.

If you've published 2 papers in top ML conferences or have a PhD in something CS related, and are interested in working with me, send me a DM.

I had a long-ish conversation with John Wentworth and want to make it known that I could probably write up any of the following distillations if I invested lots of time into them (about a day (edit: 3 days seems more likely) of my time and an hour of John's). Reply if you're really interested in one of them.

• What is the type signature of a utility function?
• Utility functions must be defined with respect to an external world-model
• Infinite money-pumps are not required for incoherence, and not likely in practice. The actual incoherent behavior is that an agent could get to states A_1 or A_2, identical except that A_1 has more money, and chooses A_2. Implications.
• Why VNM is not really a coherence theorem. Other coherence theorems relating to EU maximization simultaneously derive Bayesian probabilities and utilities. VNM requires an external frequentist notion of probabilities.