Hi, I am a Physicist, an Effective Altruist and AI Safety student/rehearser.
Ok, that makes sense. Seems like we are mostly on the same page then. I don't have strong opinions weather drawing in people via prestige is good or bad. I expect it is probably complicated. For example, there might be people who want to work on AI Safety for the right reason, but are too agreeable to do it unless it reach some level of acceptability. So I don't know what the effects will be on net. But I think it is an effect we will have to handle, since prestige will be important for other reasons. On the other hand, there are lots of people who really do want to help, for the right reason. So if growth is the goal, helping these people out seems like just an obvious thing to do. I expect there are ways funders can help out here too. I would not update much on the fact that currently most research is produced by existing institutions. It is hard to do good research, and even harder with out collogues, sallary and other support that comes with being part of an org. So I think there is a lot of room for growth, by just helping the people who are already involved and trying.
There are two basic ways to increase the number of AI Safety refreshers.1) Take mission aligned people (usually EA undergraduates) and help then gain the skills.2) Take a skilled AI researcher and convince them to join the mission.I think these two types of growth may have very different effects. A type 1 new person might take some time to get any good, but will be mission aligned. If that person looses sight of the real problem, I am very optimistic about just reminding them what AI Safety is really about, and they will get back on track. Further more, these people already exist, and are already trying to become AI Safety researches. We can help them, ignore them, or tell them to stop. Ignoring them will produce more noise compared to helping them, since the normal pressure of building academic prestige is currently not very aligned with the mission. So do we support them or tell them to stop? Actively telling people not to try to help with AI Safety seems very bad, it is something I would expect to have bad cultural effects outside just regulating how many people are doing AI Safety research.A type 2 new person who are converted to AI Safety research becasue they actually care about the mission is not to dissimilar from a type 2 new person, so I will not write more about that.However there is an other type of type 2 person who will be attracted to AI Safety as a side effect of AI Safety being cool and interesting. I think there is a risk that these people takes over the field and diverts the focus completely. I'm not sure how to stop this though since this is a direct side effect of gaining respectability, and AI Safety will need respectability. And we can't just work in the shadows until it is the right time, because we don't know the timelines. The best plan I have for keeping global AI Safety research on course, is to put as many of "our" people in to the field as we can. We have a founders effect advantage, and I expect this to get stronger the more truly mission aligned people we can put into academia. I agree with alexflint, that there are bad growth trajectories and good growth trajectories. But I don't think the good one is as hard to hit as they do. I think partly what is wrong is the model of AI Safety as a single company. I don't think this is a good intuition pump. Noise is a thing, but it is much less intrusive that this metaphor suggest. Someone at MIRI told me that to first approximation he don't read other peoples work, so at least for this person, it don't matter how much noise is published, and I think this is a normal situation, especially for people interested deep work.What mostly keep people in academia from doing deep work is the pressure to constantly publish. I think focusing on growth v.s. not growth is the wrong question. But I think focusing on deep work is the right question. So let's help people do deep work. Or, at least that what I aim to do. And I'm also happy to discuss with anyone.
I think it's great that you did this. It made the game more real, and hopefully the rest of us learned something.
I know that it is designed to guide decisions made in the real world. This does not force me to agree with the conclusions in all circumstances. Lots of models are not up to the task they are designed to deal with. But I should have said "not in that game theory situation", becasue there is probably a way to construct some game theory game that applies here. That was my bad.However, I stand by the claim that the full information game is too far from reality to be a good guide in this case. With stakes this high even small uncertainty becomes important.
What possible reason could Petrov or those in similar situations have had for not pushing the button? Maybe he believed that the US would retaliate and kill his family at home, and that deterred him. In other words, he believed his enemy would push the button.
Or maybe he just did not want to kill millions of people?
I should probably have said "we are not in that game theory situation". (Though I do think that the real world is more complex that current game theory can handle. E.g. I don't think current game theory can fully handle unknown-unknown, but I could be wrong on this point)The game of mutually assured destruction is very different even when just including known unknown.
But we are not in a game theory situation. We are in an imperfect world with imperfect information. There are malfunctioning warning systems and liars. And we are humans and not programs that get to read each others source code. There are no perfect commitments and if there where, there would be no way of verifying them.So I think that the lesson is, that what ever your public stance, and whether or not you think that there are counterfactual situation where you should nuke. In practice, you should not nuke.Do you see what I'm getting at?
From this we learn that you should not launch nukes, even if someone tells you to do it.
Applications are now closed. We'll be ~120 participants!
Me and the Helpers are finishing up the final preparations in Discord, and other places, so that we are ready to invite you in on Monday.
I've just sent out the last acceptance emails. If you have applied and not heard from me or anyone else on the organising team, then let me know asap, so we can find out what went wrong.
I may accept late applications in exchange for a bribe. (I'm actually serious about this. A *few* late applications is not a problem, and the bribe is so that you don't make a habit of it.)
Applications are now closed. We'll be ~120 participants!Me and the Helpers are finishing up the final preparations in Discord, and other places, so that we are ready to invite you in on Monday.I've just sent out the last acceptance emails. If you have applied and not heard from me or anyone else on the organising team, then let me know asap, so we can find out what went wrong.I may accept late applications in exchange for a bribe. (I'm actually serious about this. A *few* late applications is not a problem, and the bribe is so that you don't make a habit of it.)