"Here are 10 reasons against a near-term AI slow-down"

You meant 9 reasons , right?

The problem with neutral information about LLMs is that for every safety researcher trying to use it for alignment, there may be 4 more capability researchers trying to squeeze more capabilities out of it. That is, I would expect capabilities side of humanity to be collectively smarter than alignment part due to sheer scale and funding.

But it is not yet possible to fuse NNs together into a superagent either! Fusing NNs via training would be a much bigger leap out of the distribution as opposed to making them better at coordination.

This first sentence was somewhat misleading to me. Normally, I would expect "crazy" to mean literal mental disorder, not "posting controversial stuff online". Not being aware of the apparent intensity of the political sitution, I was expecting to read about Musk being hospitalized after an incident of X.

Yeah, neither can they make coffee. I would actually expect teleoperation to be possible in mostly manual situations, where the only complex part is the expert knowledge(e.g. running a given biology/chemistry experiment in a lab may require basic lab skills and very advanced chemistry knowledge).

In fact, I woud predict that a large portion of articles in natural sciences ~5 years from now, will have most of its evidence created by AI operating a bunch of bachelours in a lab.

I don't think this is misalignment. Llama is very stupid, so simple math problem may be genuinely hard for it and expert opinion may be decisive. The fact that frontier models don't do that suggests that the reason is the relative complexity of the problem. If I got a solution to a large integral and then von Neyman looked at it and said I am wrong, I would also fold fast.

Wow. I can't wait to hear more about your research!

Edited. Originally contained a stupid question

This is probably not a useful comment, but I really want someone to reassure me that I am wrong, because I don't like the view I am presenting here.

Aren't we desperate enough for an AI-supported human dictatorship?

1. Many vices of a dictatorship seem to come from insecurity about keeping power and a god-king probably does not need concentration camps if his propaganda/education/genetic engineering are good enough. Of course, being brainwashed is still bad, but not as bad as whatever happens with an unaligned AI.
2. Humans are generally good or at least have aspi

On another note, serious physical activity is something people might want to avoid while thinking hard. I remember reading that cognitive performance is weakened when brain is concerned with coordinating your movement and maintaining balance. This probably applies only to harder physical activities like lifting.

I guess we would eventually go there. Machines are going to become better friends/partners at some point, because humans are flawed. Not simply "flawed like anything real", but too flawed. They have limited time and emotional energy and so on. If the majority of humanity will spent their time with AI that is capable of making them happier/more motivated/more productive than humans would won't benefits overshadow the downsides? 

P.S. There is a creepy feeling coming from the world of people disconnected from each other, but I think it comes from the fear of the unknown(or known, but really weird like dating a rock). But I may be simply too biased.

Actually, I find this ending pretty nice. This AI does not seem to transform the universe into paperclips, it shows progress and values intelligence and as a bonus humans are still alive. A tyrant AI still seems to be human enough for me.

Or rather a derivative of the subjective progress with respect to time. In other words, effort is a derivative of progress.

Actually, I don't think that AI companions are going to have this specific flaw. If anything they will be too agreeable to what they think is your opinion. If the goal of the model is to provide a pleasant experience or a long conversation or something similar than changing someone's mind is along the worst things it can do. For example, ChatGPT often tries to identify your opinion on the specific topic and then argue in favour of it. I would expect radicalisation of society, because now everyone will be really convinced that his opinion is the best one. Only a small fraction of people that for some strange reason feels satisfied after changing its mind might actually move closer to the truth.

After reading this post, I would expect that the most probable explanation for one to wake up and see a tentacle instead of his hand will be a dream, in which such a specific tentacle is conjured because this individual fell asleep while reading this specific post

I just wanted to contribute by saying that in one of his lectures (I cannot remember the exact name), Feynman said that it is important for a physicist to know many interpretations that give the same predictions but are different computationally. The simplest example that comes to my mind is Newtonian and Lagrangian mechanics. I do not know which one is simpler in the technical sense of the word, but I am sure that every physicist is expected to know and understand both of them.