I wrote a LessWrong article that tries to estimate doubling time for a self-reproducing robot.  A critical step is that smaller robots are faster.  Most manufacturing processes scale such that they get N times faster as they get N times smaller.  I picked N=4, for reasons explained in the article.  I concluded the doubling time is five weeks.   So the time to a billion robots is on the order of five years.

Even if your goal is a human-size robot, you’re better off building small robots to build it, since they work faster.  I assumed fairly clumsy hardware, but software comparable to a human machinist in cleverness.

Nitpick: No single organism can destroy the biosphere; at most it can fill its niche & severely disrupt all ecosystems.

Have you read the report on mirror life that came out a few months ago?  A mirror bacterium has a niche of “inside any organism that uses carbon-based biochemistry”.  At least, it would parasitize all animals, plants, fungi, and the larger Protozoa, and probably kill them.  I guess bacteria and viruses would be left.  I bet that a reasonably smart superintelligence could figure out a way to get them too.

Quite right.  AI safety is moving very quickly and doesn’t have any methods that are well-understood enough to merit a survey article.  Those are for things that have a large but scattered literature, with maybe a couple of dozen to a hundred papers that need surveying.  That takes a few years to accumulate.

Could you give an example of the sort of distinction you’re pointing at?  Because I come to completely the opposite conclusion.  

Part of my job is applied mathematics.  I’d rather read a paper applying one technique to a variety of problems, than a paper applying a variety of techniques to one problem.  Seeing the technique used on several problems lets me understand how and when to apply it.  Seeing several techniques on the same problem tells me the best way to solve that particular problem, but I’ll probably never run into that particular problem in my work.

But that’s just me; presumably you want something else out of reading the literature.  I would be interested to know what exactly.

When I say Pokémon-type games, I don’t mean games recounting the adventures of Ash Ketchum and Pikachu.  I mean games with a series of obstacles set in a large semi-open world, with things you can carry, a small set of available actions at each point, and a goal of progressing past the obstacles.  Such games can be manufactured in unlimited quantities by a program.  They can also be “peopled” by simple LLMs, for increased complexity.  They don’t actually have to be fun to play or look at, so the design requirements are loose.

There have been attempts at reinforcement learning using unlimited computer-generated games.  They haven’t worked that well.  I think the key feature that favors Pokémon-like games is that when the player dies or gets stuck, they can go back to the beginning and try again.  This rewards trial-and-error learning to get past obstacles, keeping a long-term memory, and to re-plan your approach when something doesn’t work.  These are capabilities in which current LLMs are notably lacking.

Another way of saying what Claude’s missing skill is: managing long-term memory.  You need to remember important stuff, forget minor stuff, summarize things, and realize when a conclusion in your memory is wrong and needs correction.

True.  I was generalizing it to a system that tries to solve lots of Pokémon-like tasks in various artificial worlds, rather than just expecting it to solve Pokémon over and over.  But I didn’t say that, I just imagined in my mind and assumed everyone else would too.  Thank you for making it explicit!

This is an important case to think about.  I think it is understudied.  What separates current AIs from the CEO role?  And how long will it take?  I see three things:

• Long term thinking, agency, the ability to remember things, not going crazy in an hour or two.  It seems to me like this is all the same problem, in the sense that I think one innovation will solve all of them.  This has a lot of effort focused on it.  I feel like it's been a known big problem since GPT-4 and Sydney/Bing, 2 1/2 years ago.  So, by the Lindy principle, it should be another 2 1/2 years until it is solved.
• Persuasiveness.  I've known a few CEOs in my life; they were all more persuasive than average, and one was genuinely unearthly in her ability to convince people of things.  LLMs have been steadily increasing in persuasiveness, and are now par-human.  So I think scaling will take care of this.  Perhaps a year or two?
• Experience.  I don't know how much of this can be inculcated with the training data, and how much of it requires actually managing people and products and thinking about what can go wrong. Every CEO has to deal with individual subordinates, customers, and counterparties.  How much of the job is learning the ins and outs of those particular people?  Or does it suffice to have spent a thousand subjective years reading about a million people?  So we may already have this.

(Side point: As an engineer watching CEOs, I am amazed by their ability to take a few scattered hints spread across their very busy days, and assemble them into a theory of what's going on and what to do about it.  They're willing to take what I would consider intolerably flimsy evidence and act on it.  When this doesn't work, its called "jumping to conclusions", when it does work it's called "a genius move".  AIs should be good at this if you turn up their temperature.)

Eyes are exactly the worst tissue in which to express luciferase.  Lighting the eyeball from within prevents seeing things outside the eye, causing blindness.  Make your cat glow anywhere else!

Epistemic status: I know you're kidding.  So am I.

My charity extends no further than the human race.  Once in a while I think about animal ethics and decide that no, I still don't care enough to make an effort.

A basic commitment of my charity group from the beginning: no money that benefits things other than people.  We don't donate to benefit political groups, organizations, arts, animals, or the natural world.  I'm good with that.  Members of the group may of course donate elsewhere, and generally do.

We've been doing this since 1998, decades before Effective Altruism was a thing.  I don't have a commitment to Effective Altruism the movement, just to altruism which is effective.

It seems to me that I have done a lot of careful thinking about timelines, and that I also feel the AGI.  Why can't you have a careful understanding what timelines we should expect, and also have an emotional reaction to that?  Reasonably coming to the conclusion that many things will change greatly in the next few years deserves a reaction.