Quick Takes

Some meandering thoughts on alignment A nearcast of how we might go about solving alignment using basic current techniques, assuming little/no substnative government intervention is: 1. During the beginning of takeoff, we do control, attempting to prevent catastrophic actions (e.g. major rogue internal deployments (RIDs)), while trying to elict huge amounts of AI labour. 2. At some point, the value of scaling AI capabilities while trying to maintain control will be very low, because the main bottleneck on elictation of useful AI labour is human oversight. If you are pausing the AI all the time to wait for the humans to understand what the AIs are doing, it doesn't help to make the AI smarter. At this point AI companies pause / slowdown as much as possible (e.g. unilaterally or via coordinating with other companies), but this isn't for very long. During the pause, try to build an AI system that you would trust to manage the intelligence explosion. 3. Handoff to the AI system, i.e. allow the AI to manage the training run, conduct large, open ended experiments, etc, that humans cannot effectively oversee. The AIs should be corrigible and attempt to ensure that their successors are corrigibile, and still check in with humans as much as possible post-handoff, keep them informed about the situation. At this point, it's correct to handoff to the AIs because the alternative -- scaling to vastly superhuman AI without letting the human-slightly superhuman AIs do a huge amount of reserach -- is even more dangerous. My best guess (overly detailed story) for how this plan goes if you try to implement it in the real world is: * High-stakes control basically works (during steps 1-2). We prevent AIs from doing major RIDs or subsantially subverting safety/ human oversight. * Elicitation will totally fail on open-ended, hard to check tasks such (like preparadigmatic agent foundations reserach, etc). We'll get some useful work on the more prosaic agendas, e.g. misalignment an

Projecting AI Water Usage. Environmentalists warn that each 100-word AI prompt is estimated to use roughly one bottle of water, and large data centers can consume up to 5 million gallons per day — equivalent to the needs of a town of 10,000 to 50,000 people. On the other hand, Andy Masley argues, "On the national, local, and personal level, AI is barely using any water, and unless it grows 50 times faster than forecasts predict, this won’t change." And Bentham's Bulldog writes, "The environmentalist case against AIcompletely falls apart upon even cursory examination of the facts." I offer my own projection: AI will eventually consume 10^41 – 10^47 gallons of water — up to a hundred trillion trillion times the water in Earth's oceans. How much water is out there? Water is the third most abundant molecule in the universe, after H2 and CO. The Solar System alone contains ~10^26 kg of water in planets, moons, and comets — about 100,000 times Earth's oceans (Kotwicki 1991). There are ~10^11 stars in the Milky Way, each plausibly endowed with a similar complement of icy bodies. And the galaxy's molecular clouds contain vast reservoirs of water ice on dust grains. A reasonable estimate for total water in the Milky Way is a few x 10^37 kg — about 10^16 Earth-oceans. How many galaxies does the AI get? The cosmic event horizon — the boundary beyond which even light-speed travel cannot reach, due to the accelerating expansion of space — sits at roughly 16.5 billion light-years (Ord 2021). This encloses about 20 billion galaxies, roughly 5% of the observable universe. However, the future AI may bump into competitors before reaching the cosmic event horizon. Robin Hanson's grabby aliens model estimates that each "grabby civilization" — one that expands at a significant fraction of light speed and visibly transforms its territory — would eventually control 10^5 to 3 x 10^7 galaxies before meeting others. The water budget of future AI. Putting it together, with ~10^37 kg of w

Rob Wiblin asked: [...] My response: [...] As additional context, I'll note that I was just talking about misalignment risk and security; I didn't mention considerations relevant to e.g. preventing coups.

I don't feel confused by LLMs seeming very smart while being unable to automate hard work. Sometimes people find it mysterious or surprising that current AIs can't fully automate difficult tasks given how smart they seem. I don't find this very confusing. Current LLMs are just not that "smart" (yet). They compensate using very broad knowledge and strong heuristics that are mostly domain-specific. In other words, they have high crystallized intelligence but lower fluid intelligence. In humans, crystallized and fluid intelligence are very correlated due to limited time for learning and limited capacity for knowledge/memorization, but AIs can train for longer and (for unclear reasons) seem to have much higher capacity for knowledge/memorization. So, AI capabilities are often overestimated by naive comparisons. [1] I do find it somewhat mysterious/surprising that humans have such poor memory despite seemingly having so many parameters, and that some people have much better memory than others. My view is that this crystallized vs fluid distinction is maybe the first principal component of human vs AI differences and this explains a lot of the differences, but not all of them. We can add an additional "component" to our analysis of fluid intelligence by distinguishing between short-run fluid intelligence (we could call this "adaptation") and longer term fluid intelligence (this might effectively be sample efficiency of longer run learning). I think AIs are relatively stronger at short-term fluid intelligence than long-term fluid intelligence. To be clear, this doesn't mean I think extremely powerful AI isn't near: I expect full AI R&D automation [2] in <5.5 years [3] (and it seems 25% likely in <2.5 years). These AIs will likely retain an inhuman profile but will be trained heavily on relevant tasks (while still having much stronger fluid intelligence than current AIs). But pretty soon (0.25-4 years) after full AI R&D automation, I do expect AIs to exceed humans at e

AI doesn't have an individual existence like a human-like organism, and we shouldn't change that unless we want to face enormous ethical questions. We might already be moving in that direction, however. 1. Organisms have a clearly bounded, independent physical existence for most of their lives. LLMs don't have a clearly defined physical existence that maps well to the mental persistence they do have. Treating chat sessions as the units of continuous individual mental activity, many sessions run on the same hardware, and they can be stopped, restarted on different hardware, cloned, etc. Even with robotics, the inference is rarely on-device. 2. An organism's cognition is a self-modifying function; memories, habits, etc. are encoded persistently in the brain's wiring. LLMs mainly use the context window to emulate this, but this is finite, unlike rewriting your own weights. I think the phenomenon of every adaptation layering over time contributes significantly to the notion we have of an individual organism. 3. Organisms are "trained" on first-personal data. I learned how to speak English based on my own "sensor data," and my knowledge of Tolstoy's Confessions comes from when I picked up a physical object, turned the yellowed pages with my hands, and then discussed it in a classroom on a late fall afternoon. It's not like the tokens were beamed directly into my mind. This constant background context produces the notion of the self organically. 4. Organisms are continuously acting and continuously taking in stimulus. There are no discrete conversational turns between an organism and its surroundings. 5. Organisms reproduce independent of other species by using the physical bodies from part 1. But all of these are blurry, and many are already eroding: 2. I suspect that much of human brain-update processing is amortized using sleep. If so, an analogy can be made to model training as a sort of long-term sleep, especially if data from model deployment is used. 3. Training

There's a strong pattern in ratfic of the protagonist "winning" by gaining the power to design a new world order from scratch—i.e. taking over the world. It's a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it's hard to unsee. You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn't have to be that bad! The writers chose to design them that way—I expect in significant part because that provides a narratively compelling backdrop for the thing they wanted to write about, which was their heroes taking over the world. From my current perspective, this feels ominous in a kinda similar way as if I read a bunch of Marxist fiction from the 1800s and noticed that it always ended in a communist revolution. Spoilers for a range of ratfic (unfortunately telling you which ratfic itself constitutes a spoiler): EDIT: the spoiler tags seem not to be working on the images. Mods, can you help? For now I'll add a bunch of space below. Expect many spoilers if you scroll down. And some examples which (at least partially) subvert this trope:

survey: what brand of melatonin do you use? i want to run an experiment on melatonin degradation using the most popular brand of melatonin.