Engineer at Donor to LW 2.0.

Wiki Contributions


Taking Clones Seriously

Maybe the average intelligence of adults who came from von Neumann embryos would be substantially greater than from the average population, such that the extremely intelligent ones occur a hundred times as often as usual. That's probably the most optimistic outcome.

I think this is actually a quite pessimistic outcome and that IQ is much more heritable (and genetically determined) than this assumes.

Yudkowsky and Christiano discuss "Takeoff Speeds"

Seems like there's a spectrum between smooth accelerating progress and discontinuous takeoff. And where we end up on that spectrum depends on a few things:

  • how much simple improvements (better architecture, more compute) help with a wide variety of tasks
  • how much improvements in AI systems is bottlenecked on those tasks
  • how many resources the world is pouring into finding and making those improvements

Recent evidence (success of transformers, scaling laws) seems to suggest that Eliezer was right in the FOOM debate that simple input changes could make a large difference across a wide variety of tasks.

It's less clear to me though whether that means a local system is going to outcompete the rest of the economy, because it seems plausible to me that the rest of the economy is also going to be full-steam ahead searching the same improvement space that a local system will be searching.

And I think in general real world complexity tends to smooth out lumpy graphs. As an example, even once we realize that GPT-2 is powerful and GPT-3 will be even better, there's a whole bunch of engineering work that had to go into figuring out how to run such a big neural network across multiple machines.

That kind of real-world messiness seems like it will introduce new bottlenecks at every step along the way, and every order-of-magnitude change in scale, which makes me think that the actual impact of AI will be a lot more smooth than we might otherwise think just based on simple architectures being generally useful and scalable.

Yudkowsky and Christiano discuss "Takeoff Speeds"

But after the 10^10 point, something interesting happens: the score starts growing much faster (~N). 

And for some tasks, the plot looks like a hockey stick (a sudden change from ~0 to almost-human).


Judging the preliminary results, the FOOM could start like this:

"The GPT-5 still sucks on most tasks. It's mostly useless. But what if we increase parameters_num by 2? What could possibly go wrong?"



  • doing things in the real world requires diverse skills (strong performance on a diverse set of tasks)
  • hockey-sticking performance on a particular task makes that task no longer the constraint on what you can accomplish
  • but now some other task is the bottleneck
  • so, unless you can hockey-stick on all the tasks all at once, your overall ability to do things in the world will get smoothed out a bunch, even if it still grows very rapidly
Yudkowsky and Christiano discuss "Takeoff Speeds"

bet on whether there will be a yearly GWP/GDP that exceeds 110% of a previous year

Did you mean "that exceeds 110% of all previous years"? (To exclude steady growth that eventually goes over 110% in aggregate, like [1.0, 1.03, 1.06, 1.09, 1.12].)


Ah, got it. Thanks!

May I ask for some help getting started with crypto?

If you want exposure to crypto, I'd say: put a small fraction of your portfolio in the top coins on, and don't try to do anything fancier than that.

I Really Don't Understand Eliezer Yudkowsky's Position on Consciousness

Where I disagree is that we 100% need a separate "information processing" and "inner listener" module.

I didn't understand this part. Do you mean that EY thinks we need these two modules and you don't think that, or the other way around?

(I think this is a generic problem that arises pretty much whenever someone uses this kind of phrasing, saying "Where I disagree is that X". I can't tell if they're saying they believe X and the person they disagree with believes not-X, or the other way around. Sometimes I can tell from context. This time I couldn't.)


The by-line seems a little confusing on this one. It was written by Chris Olah, and posted by abergal and Nick_Beckstead acting together?

Ruling Out Everything Else

Relatedly, I think an often underappreciated trick is just to say the same thing in a couple different ways, so that listeners can triangulate your meaning. Each sentence on its own may be subject to misinterpretation, but often (though not always) the misinterpretations will be different from each other and so "cancel out", leaving the intended meaning as the one possible remaining interpretation.

I have it as a pet peeve of mine when people fail to do this. An example I've seen a number of times — when two people with different accents / levels of fluency in a language are talking (e.g. an American tourist talking to hotel staff in a foreign country), and one person doesn't understand something the other said. And then the first person repeats what they said using the exact same phrasing. Sometimes even a third or more times, after the listener still doesn't understand.

Okay, sure, sometimes when I don't understand what someone said I do want an exact repetition because I just didn't hear a word, and I want to know what word I missed. And at those times it's annoying if instead they launch into a long-winded re-explanation.

But! In a situation where there's potentially a fluency or understanding-of-accents issue, using the exact same words often doesn't help. Maybe they don't know one of the words you're using. Maybe it's a phrasing or idiom that's natural to you but not to them. Maybe they would know the word you said if you said it in their accent, but the way you say it it's not registering.

All of these problems are solved if you just try saying it a different way. Just try saying the same thing three different ways. Making sure to use different (simple) words for the main ideas each time. Chances are, if they do at least somewhat speak the language you're using, they'll pick up on your meaning from at least one of the phrasings!

Or you could just sit there, uncreatively and ineffectively using the same phrasing again and again, as I so often see...

Load More