That one is interesting! Where do you take probabilities from?

Could you do it as a text post with short explanations of each?

I think yes, it would help to avoid confusion. 

Have a sufficient financial safety net

I think, this condition is important only if I am going to leave my full-time job and switch to unpaid  AI Safety projects. For some people (who have financial security), this may be the case. Many, including myself, do not have this security. It does not mean I can't do any projects until I get enough funds to survive. Rather, it means that I can do only part-time projects (for me, it was organising mentoring programs and leading AI Safety Camp project). Meanwhile, I still think applying to the roles that seem to be a good fit for me makes quite a lot of sense - I would rather spend 40 hours/week working on AI Safety than on a regular job. Maybe it should be something like 80% projects, 20% applying (the numbers are random). 


I feel that the percentage of people who can afford not to have paid work and only do AI Safety projects till AGI arrives is not that high. It would be nice to have also a strategy and recommendations, what a person can do for AI Safety with 10 hours/week, or 5, or even 1. I think the boundary where one can do something useful is quite low - even with 5 minutes/week they can e.g. repost stuff in social networks. 



 

1. The idea of total extinction might seem wild for non-nerds. Maybe it is good to start with small things:
   -the job you are doing will be done by AI 
   -whatever education you or your kids get in college, it won't give you a job
   -even if you are working on AI with AI you still can be replaced completely by AI
   
   This may at least make them to think more about impact on society and importance of the problem on the gut level, and from there we could go to more serious issues
2. I would say the nuclear war would be the least sci-fi scenario. Arm race leads to using AI everywhere to beat the opponent, including systems responsible for observing and responding to opponent's missile strikes, and then it goes rogue.
3. I think bioweapons can be persuasive.  We know that there are viruses like smallpox with very high lethality and very big virulence. Actually COVID and smallpox would be a good starting point for explanation. I would say something like: "Remember COVID spread everywhere despite all restrictions? Its incubation period was roughly 2 weeks. For smallpox, it can be like 40 days. It is incredibly viral.  In a modern world, in 40 days it will be everywhere, and then it is too late. Lethality rate 50-80%. And for smallpox we have a vaccine, that is why we are safe against a smallpox. AI already designs viruses and can easily be used to design something like smallpox, but for which this vaccine does not work".      

Thank you very much for catching the mistake! I checked, you are completely right. 

I don't think they passed it in a full sense. Before LLM, there was a 5 minute Turing test, and some chatbots were passing it. I think 5 minutes is not enough. I bet that if you give me 10 hours, any currently existing LLM and human, we will communicate only via text, I will be able to figure out who is who (if both will try hard to persuade in their humanity). I don't think LLM can come up yet with a consistent non-contradicting life story. It would be an interesting experiment :)

Would you mean similarity on the outer level (e.g. Turing test) or at inner (e.g. neural network structure should resemble brain structure?
If the first - would it mean that when AI passes Turing test it would be sentient?
If the second - what are the criteria for similarity?  Full brain emulation or something less complicated?

1. Are you working with SOTA model? Here, mathematicians report a quite different story  https://www.scientificamerican.com/article/inside-the-secret-meeting-where-mathematicians-struggled-to-outsmart-ai/
2. I guess "good at" was improper wording. I did not mean that they do not produce nonsense. I meant that sometimes they can produce a correct solution. It is like the person may be not fit for running 100 meters in 10 seconds every day, but even if they do it in 5% of cases this is already impressive, and shows that it is possible in principle. And I guess "Ph.D. level" sounded like they can write a Ph.D. thesis from scratch. I just meant that there are short well-formulated problems that would require Ph.D. student a few hours, if not few days, which current LLM can solve in non negligible number of cases.    

Can you expand your argument why LLM will not reach AGI? Like, what exactly is the fundamental obstacle they will never pass? So far they successfully doing longer and longer (for humans ) tasks https://benjamintodd.substack.com/p/the-most-important-graph-in-ai-right
I neither can see why in a few generations LLM won't be able to run a company, as you suggested. Moreover, I don't see why it is necessary to get to AGI. LLM are already good at solving complicated, Ph.D. level mathematical problems, which improves. Essentially, we just need an LLM version of AI researcher. To create ASI you don't need a billion of Sam Altmans, you need a billion of Ilya Sutskevers. Is there any reason to assume LLM will never be able to become an excellent AI researcher?