As you say, we reason too much with our current knowledge, like every society in the past that thought it understood everything about the universe.

Some people suggest Dyson spheres and Von Neumann probes, but an advanced AI could very well find these inventions unnecessary to build and imagine many other things to prioritize goals that we don't know yet.

#4. Can't cooperate to avoid AGI

Maybe we can. This is how the Montreal Protocol came about: scientists discovered that chlorofluorocarbons were bad for the ozone. Governments believed them, then the Montreal Protocol was signed, and CFC use fell by 99.7%, leading to the stabilization of the ozone layer, perhaps the greatest example of global cooperation in history.

It took around 15 years from the time scientists discovered that chlorofluorocarbons were causing a major problem to the time the Montreal Protocol was adopted.

How can scientists convince the world to cooperate on AGI alignment in less time?

I just read your linked post. In the comments someone proposes the idea that computing will migrate to the next level of abstraction. This is the idea I was quoting in my post, that there will be fewer hackers, very good at tech, and more idea creators who will run IAs without worrying about what's going on under the hood.
I agree with your point that 1% error can be fatal in any program and that what is coded by an AI should be checked before implementing the code on multiple machines.

Speaking of which, I'm amazed by the fact that Chat-GPT explains in common language most of the code snippets. However, my knowledge in programming is basic and I don't know if some programming experts managed to make Chat-GPT perplexed by a very technical, very abstract code snippet.

Yes, I hadn't thought about the fact that you have several books to your credit and so people know that you have writing skills. The public can trust you because they know your past work.

Yes, human beings could easily question a person's creations by asking about their technique, their writing choices, their artistic choices, their aspirations. I hadn't thought of that.

I'm trying to predict what's going to happen in the very near future but it's very difficult. In 2012 we thought we would have autonomous cars in 2022 and that the tools of creativity were very hypothetical. The opposite has happened.

I had a similar discussion with a tattoo artist two days ago. Tattoo machines will exist, but some people will prefer to be tattooed by an artist because of his style, his talent, and his humanity. You can prove that you are a human tattoo artist by tattooing the client, so AI is not a problem here.
As for the writings produced by a human being, I wonder how you can prove to the readers that you are the author of your writings and not an artificial intelligence. I wonder the same thing about digital pictures or musical composition.

Sure, you can do things for your own enjoyment, even if an AI automates them.

Maybe as a writer, you'll have to spend some time in a room with cameras when you're writing so that everyone can prove that you wrote your content and not an AI.

Live arts performances could become much more common, just so people can say it's art made entirely by humans.

Edit: English grammar.

Thank you for your advice, I have modified the question and the attached text.