Every person and his/her descendants get eternal youth
This has worse implications than you might expect.
If everyone can have unlimited descendants and reproduce frequently, the population could, for example, double every 20 years. In this scenario the human population would be over 10^150 in 10 thousand years, but the observable universe only has about 10^80 atoms.
Notice the only problem here is unlimited reproduction, but without breaking multiple physics law's (at least constant energy and the speed of light) we are not allowed to have children freely.
I'm pretty sure the usual impression of IC people have is "if I had no morals, values, beliefs or anything else and I had a task, I would do all power-seeking behaviours possible and that wouldn't end well for people around me", and then you just scale this to something that is thousands of times more capable than a human and things end pretty bad for humanity. Will AI not be like that? Maybe, but right now they are being trained to complete long-term tasks, and they cheat a lot. Worth noting that most simple tasks that we think of, like maximizing papercl... (read more)
My family, friends, and colleagues have faithfully adhered to this strategy while I’ve tried explaining that AI poses a major existential threat. All they hear is "wah wah wah wah AI might kill us all", and they reply that we should focus on what we can control, or that it just sounds absurd.
Ignoring what other people have to say won't work simply because you believe you are right or because the opposing argument is too complex (e.g., 10 years ago, the argument that AI would become an existential threat was too weird and complex for most, but was right any... (read more)
It’s like cars: the solution to reducing accidents is to make cars and roads safer, not banning cars nor just deciding that car manufacturers are evil. Solutions can never be perfect, but they put us in a better place on the risk-benefit trade-off curve.
Yann LeCun couldn't have used a worse example than cars here.
It's not as if the car industry were lobbying and campaigning against the public good in the 1920s by promoting the term "jaywalking" to shift the blame for early pedestrian deaths and injuries away from their new products.
Nor is it like there is ... (read more)
I feel like "Big-World Intuitions" implies that the main breaking point of those intuitions is the size of the environment and this can be misleading. In one of the examples given: "share scientific knowledge freely" the intuition might break down even if you are in a Big-World situation. You might discover information hazards accidentally anyway. For those cases, it is better to share with specific institutions first before revealing to the wider public. Similarly, in the chess example: how you develop in the opening might depend on what your opponent is ... (read more)
Why are yours and most other predictions for take-off timelines years long? My personal median for take-off via RSI (from a reliable auto-coder to ASI) is a single day. The primary reasons I expect it to be faster:
With unlimited reproduction, it could get to the point that dying in the first place was the best option. But no, death is worse than living forever usually. Sorry if I was unclear, my comment was just meant to be a reminder that even a post-scarcity world might have some restrictions (not having children could be one of them) and one badly chosen post-scarcity world might end up ugly. I would press the button BTW.