Wiki Contributions


Biology-Inspired AGI Timelines: The Trick That Never Works

It's very easy to construct probability distributions that have earlier timelines, that look more intuitively unconfident, and that have higher entropy than the bio-anchors forecast. You can just take some of the probability mass from the peak around 2050 and redistribute it among earlier years, especially years that are very close to the present, where bioanchors are reasonably confident that AGI is unlikely.

Biology-Inspired AGI Timelines: The Trick That Never Works

+1. I will also venture a guess that:

OpenPhil: Well, search by evolutionary biology is more costly than training by gradient descent, so in hindsight, it was an overestimate. Are you claiming this was predictable in foresight instead of hindsight?

is a strawman. I expect that the 2006 equivalent of OpenPhil would have recognised the evolutionary anchor as a soft upper bound. And I expect current OpenPhil to perfectly well understand the reasons for why this was predictable in foresight.

Christiano, Cotra, and Yudkowsky on AI progress

Sam Altman explicitly contradicted that in a later q&a, when someone asked him about that quote.

Omicron Variant Post #1: We’re F***ed, It’s Never Over

A factor 1.5-3 of that could be immune erosion, in which case the R0 would be more like 10-20. And more importantly, I don't know anything that contradicts Zvi's intuition that this little data shouldn't push us far away from our priors.

Omicron Variant Post #1: We’re F***ed, It’s Never Over
I think this final graph is a bit confused here, unless ‘the original strain’ here means Delta. Delta had about a 120% advantage over ‘the original strain’ or 70% over Alpha. I’m going to take this to mean 500% as compared to that 120%, so 600% of original versus 220% of original, or about a 170% additional increase. Which is… better, but still quite a lot.

I'm pretty sure the graph is trying to say 500% over delta. If you run the numbers on a very similar graph, you get that the new disease is slightly above 5.5 times as infectious as delta, which you could round to a 500% increase. (Assuming identical generation length. Also, I haven't looked into sources for the numbers.)

South Africa’s vaccination rate is sufficiently low, and this rate of spread so high, that it wouldn’t much matter if there was vaccine escape properties, although it would presumably matter if there was escape from natural immunity.

I don't know why you expect a large difference there. I'd guess that roughly equal numbers of people in south africa has been vaccinated as has natural immunity.

Or, okay, I can see two reasons to expect natural immunity to be a bigger deal:

  • natural immunity is more common in the most exposed part of the population.
  • there's more uncertainty about the amount of natural immunity in south africa, and if south africa has very sizeable natural immunity and there's substantial immunity erosion, that would be a pretty good explainer for why omicron would be spreading so much faster than delta. So there's some bayesian update towards south africa having sizeable natural immunity.

Don't know where this nets out.

Christiano, Cotra, and Yudkowsky on AI progress

To be clear: Do you remember Sam Altman saying that "they’re simply training a GPT-3-variant for significantly longer", or is that an inference from ~"it will use a lot more compute" and ~"it will not be much bigger"?

Because if you remember him saying that, then that contradicts my memory (and, uh, the notes that people took that I remember reading), and I'm confused.

While if it's an inference: sure, that's a non-crazy guess, and I take your point that smaller models are easier to deploy. I just want it to be flagged as a claimed deduction, not as a remembered statement.

(And I maintain my impression that something more is going on; especially since I remember Sam generally talking about how models might use more test-time compute in the future, and be able to think for longer on harder questions.)

Rapid Increase of Highly Mutated B.1.1.529 Strain in South Africa

Can you say more about what this picture depicts and why it's relevant?

Rapid Increase of Highly Mutated B.1.1.529 Strain in South Africa

Quick googling, numbers for south africa:

  • Population 59 million.
  • 28% of population at least one dose, 24% fully vaccinated.
  • 89,771 deaths. At 0.5% fatality rate that would be 18 million people ~= 30% of the population.

So maybe 100% immune escape would be a factor 1.5-3? Leaving at least a factor 2-3 for generally increased infectiousness. (Assuming unchanged generation length.)

Yudkowsky and Christiano discuss "Takeoff Speeds"
Oh, come on. That is straight-up not how simple continuous toy models of RSI work. Between a neutron multiplication factor of 0.999 and 1.001 there is a very huge gap in output behavior.

Nitpick: I think that particular analogy isn't great.

For nuclear stuff, we have two state variables: amount of fissile material and current number of neutrons flying around. The amount of fissile material determines the "neutron multiplication factor", but it is the number of neutrons that goes crazy, not fissile material. And the current number of neurons doesn't matter for whether the pile will eventually go crazy or not.

But in the simplest toy models of RSI, we just have one variable: intelligence. We can't change the "intelligence multiplication factor", there's just intelligence figuring out how to build more intelligence.

Maybe exothermic chemical reactions, like fire, is a better analogy. Either you have enough heat to create a self-sustaining reaction, or you don't.

Yudkowsky and Christiano discuss "Takeoff Speeds"

This is my take: if I had been very epistemically self-aware, and carefully distinguished my own impression/models and my all-things considered beliefs, before I started reading, then this would've updated my models towards Eliezer (because hey, I heard new not-entirely-uncompelling arguments) but my all-things considered beliefs away from Eliezer (because I would have expected it to be even more convincing).

I'm not that surprised by the survey results. Most people don't obey conservation of expected evidence, because they don't take into account arguments they haven't heard / don't think carefully enough about how deferring to others works. People will predictably update toward a thesis after reading a book that argues for it, not have a 50/50 chance of updating positively or negatively on it.

Load More