Denis Drescher

I’m working on Impact Markets – markets to trade nonexcludable goods.

If you’re also interested in less directly optimific things – such as climbing around and on top of boulders or amateurish musings on psychology – then you may enjoy some of the posts I don’t cross-post from my blog, Impartial Priorities.

Pronouns: Ideally they or she, but the others are fine too.

Wiki Contributions


MIRI announces new "Death With Dignity" strategy

Agreed. Also here’s the poem that goes with that comment:

Do not go gentle into that good night,
Old age should burn and rave at close of day;
Rage, rage against the dying of the light.

Though wise men at their end know dark is right,
Because their words had forked no lightning they
Do not go gentle into that good night.

Good men, the last wave by, crying how bright
Their frail deeds might have danced in a green bay,
Rage, rage against the dying of the light.

Wild men who caught and sang the sun in flight,
And learn, too late, they grieved it on its way,
Do not go gentle into that good night.

Grave men, near death, who see with blinding sight
Blind eyes could blaze like meteors and be gay,
Rage, rage against the dying of the light.

And you, my father, there on the sad height,
Curse, bless, me now with your fierce tears, I pray.
Do not go gentle into that good night.
Rage, rage against the dying of the light.

I totally empathize with Eliezer, and I’m afraid that I might be similarly burned out if I had been trying this for as long.

But that’s not who I want to be. I want to be Harry who builds a rocket to escape Azkaban, the little girl that faces the meteor with a baseball bat, and the general who empties his guns into the sky against another meteor (minus all his racism and shit).

I bet I won’t always have the strength for that – but that’s the goal.

Microcovid Becoming Less Useful

Hi! I suppose that question mostly goes to Adam? The importer is fixed, so I’m not doing any of this anymore. What I did was to extrapolate to the current, incomplete week n using the slope from week n-2 to n-1. Then I set the percent increase to 0 because the week is actually the current week already.

But the differences from week to week will, in most cases, be minor, so I don’t think it’s important to get this exactly right. There are so many other uncertain factors that go into the model that I don’t recommend investing too much time into this one.

Would (myopic) general public good producers significantly accelerate the development of AGI?

My friend Rai said in a Telegram chat:

So my thinking is something like: If you just throw money at FOSS ML libraries, I expect you'd mostly shorten the time from "some researcher writes a paper" and "that model is used in a billion real-world projects". I think you'd make whatever AI tech exists be more broadly distributed. I don't think that would directly make stronger AI arrive faster, because I think it would mostly just give people lots of easy-to-use boxes, like when CNNs got popularized, it became quite trivial to train any sort of visual classification task you wanted.

But I can see that this would make weaker AI get deployed faster, so that gains from capabilities research could be easier to see for investors, so it could indirectly pour more effort into developing stronger capabilities.

The change of timelines that would produce would depend on to what degree you believe that we are in a "deployment overhang", in the sense of "practical deployment of models is lagging far behind what we have capacities for". I can see arguments for both why we are and aren't in such an overhang. Argument for would be of the shape "have you seen the stuff GPT-3 can do? Why doesn't everyone have a competent virtual assistant yet?" An argument against would be "look at how slow we're going with e.g. self-driving cars or AI in medicine, capabilities research just glosses over lots of stuff that blocks practical deployment".

I think it also depends a lot on what exactly you'd be funding. For example, "cutting-edge pre-trained deep RL for everyone" versus "explainable interpretable robust models for everyone"


Woot, thanks! Can I share that with the others, including the author of the question? My interpretation was even that it’s about libraries like cchardet – super fundamental ones. Lots of websites misdeclare their encodings, so you get a lot of messy text with encoding errors when you try to read it. Make cchardet a bit more magical, add support for a few more encodings, and you can extract a bit more training data, or a developer saves a day of hard-coding the right encodings to use for a bunch of domains. That, except with thousands and thousands of libraries like cchardet.


Sure you can share that, even as a comment with attribution to me if you think it could be useful.

With stuff like more fundamental libraries there it becomes i think a question of generally speeding up industry/innovation. I think you'd probably want to plug all this into a general development model.

Microcovid Becoming Less Useful

Good. There’s a site for Switzerland like that too. I extrapolated from that in a similar manner. :-)

Microcovid Becoming Less Useful

Thanks for the analysis!

Some of these points seem like attempts at being conservative (I imagine most people will prefer to err in the direction of caution when advising others), while other points just look like the project is not well-maintained anymore.

Do you have a feeling for what a good adjustment factor is that one could apply to the result to compensate the first kind of problem, so everything related to others’ vaccination status? (I’ll just enter the local data manually while the importer is broken.)

(I’m hesitant to go with a budget that would imply a 50% yearly risk for me, but if I’m going to stick with my approximate 3% p.a. risk budget, I want to do so using a median estimate, not a conservative one. Update: Did some math. I think I’m comfortable with a 20% p.a. budget now. But I don’t really do budgets because I weigh for every event what risk it is worth or whether there’s an equally good but safer alternative.)

Reviews of “Is power-seeking AI an existential risk?”

That’s a great report and exercise! Thank you!

I hopelessly anchored on almost everything in the report, so my estimate is far from independent. I followed roughly Nate’s approach (though that happened without anchoring afaik), and my final probability is ~ 50% (+/- 5% when I play around with the factors). But it might’ve turned out differently if I had had an espresso more or less in the morning.

50% is lower than what I expected – another AI winter would lead to a delay and might invalidate the scaling hypothesis, so that the cumulative probability should probably increase more slowly after 2040–50, but I would’ve still expected something like 65–85%.

My biggest crux seems to be about the necessity for “deployment.” The report seems to assume that for a system to become dangerous, someone has to decide to deploy it. I don’t know what the technology will be like decades from now, but today I test software systems hundreds of times on my laptop and on different test systems before I deploy them. I’m the only user of my laptop, so privilege escalation is intentionally trivial. As it happens it’s also more powerful than the production server. Besides, my SSH public key is in authorized_keys files on various servers.

So before I test a software, I likely will not know whether it’s aligned, and once I test it, it’s too late. Add to that that even if many people should indeed end up using carefully sandboxed systems for testing AGIs within a few decades, it still only takes one person to relax their security a bit.

So that’s why I merged the last three points and collectively assigned ~ 95% to them (not anchored on Nate afaik). (Or to be precise I mostly ignored the last point because it seems like a huge intellectual can of beans spanning ethics, game theory, decision theory, physics, etc., so more than the purview of the report.)

Reviews of “Is power-seeking AI an existential risk?”

Maybe one should be more hesitant to assign very low probabilities to factors in a conjunction than very high probabilities because the extreme of 100% probability should not be more controversial than omitting a factor.

With a bit of creativity someone could probably come up with dozens of verisimilar additional factor along the lines of “and false vacuum decay hasn’t happened yet.” If we then have to be humble and assign (say) no more than 99% to each of those, it just takes 10 noisy factors to bias the estimate down to ~ 90%.

In this case I went with Nate’s approach and merged the last three factors. None of them seemed silly to me, but I felt like splitting them up didn’t help my intuitions.

How to Be Happy

Improve your self-esteem and optimism. This is tricky. First, too much self-esteem can lead to harmful narcissism.

A friend of mine uses a great trick here. She has very high self-esteem (though maybe not higher than the esteem that I hold her in) but doesn’t come across narcissistic at all because she’s also very warm and empathetic with a lot of care and curiosity for the people around her. These multiply and are therefore particularly uplifting for those around her: I (for example) not only feel interesting but I feel so interesting that I’m interesting to a person who could find an incredibly interesting person in any random mirror, which, I surmise, must be a very high bar.

So maybe it makes senses to train extraversion, empathy, social skills, and self-esteem all at the same time.

My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)

I just want to send you some sympathy this way. Everything you’ve gone through and all the self-doubt and everything else that I can’t put a name to must be very stressful an exhausting. Reading and responding to hundreds of comments, often very critical ones, is very exhausting too. And who knows what else is going on in your life at the same time. Yet your comments show none of the exhaustion that I would’ve felt in your situation.

I’d also like to second what Rafael already said!

It seems it’s been a few weeks since most of these discussions happened, so I hope you’ve had a chance to relax and recover in the meantime, or I hope that you’ll have some very soon!

How might cryptocurrencies affect AGI timelines?

Oh sorry, I didn’t see your comment.

  1. There is the existence proof of gold, silver, platinum, etc. But people have mostly convinced me already that 2 OOM are not going to be enough to have much of an effect.

  2. I think you understand my argument correctly. :-) You argued that AI safety will not differentially speed up compared to AI capabilities because it will slow down at the same rate. Maybe not literally the same rate but the same rate in expectation. But that still seems unlikely to me since I can think of reasons why it would slow down less but can’t think of reasons why it would slow down more.

Load More