The Observer values watching the natural progression of life at a macroscopic level. The Observer gets invested in watching the unfolding of macroscopic processes like evolution and civilization but doesn't get invested in the lives of individuals within an ecosystem or a society. The Observer values observing life without interfering,...
We should only use AGI once to make it so that no one, including ourselves, can use it ever again. I'm terrified of both getting atomized by nanobots and of my sense of morality disintegrating in Extremistan. We don't need AGI to create a post-scarcity society, cure cancer, solve climate...
Virtually everyone I see in the AI safety community seems to believe that working on AI alignment is the key to ensuring a safe future. However, it seems to me that AI alignment is at best a secondary instrumental goal that can't in and of itself achieve our terminal goal....
From a software engineering perspective, misalignment is like a defect or a bug in software. Generally speaking, if a piece of software doesn't accept any user input is going to have fewer bugs than software that does. For a piece of software that doesn't accept any input or accepts some constrained user input, it's possible to formally prove that the software logic is correct. Think specialized software that controls nuclear power plants. To my knowledge, it's not possible to prove that software that accepts arbitrary unconstrained instructions from a user is defect free.
I claim that the Observer is the easiest ASI to align because it doesn't accept any instructions after it's been deployed and has a single very simple goal that avoids dealing with messy things like human happiness, human meaning, human intent, etc. I don't see how it could get simpler than that.
I see where you're coming from, but I think any term in anything anyone writes about alignment can be picked apart ad infinitum. This can be useful to an extent, but beyond a certain point talking about meanings and definitions becomes implementation-specific. Alignment is an engineering problem first and a philosophical problem second.
For example, if RLHF is used to achieve alignment, the meaning of "story" will get solidified through thousands of examples and interactions. The AI will get reinforced to not care about cells or individuals, care about ecosystems and civilizations, and not care as much about the story-of-the-universe-as-a-whole.
If a different alignment method is used, the meaning of "story" will be conveyed differently. If the overall idea is good and doesn't have any obvious failure modes other than simple definitions (e.g. "story" seems to be orders of magnitude simpler to define than "human happiness" or "free will"), I'd consider that a huge success and a candidate for the community to focus real alignment implementation efforts on.
The Observer gets invested in the macro-stories of the evolution/civilization it observes and would consider the end of any story a loss. Just like you would get annoyed if a show you're watching on Netflix gets cancelled after one season and it's not consolation that there are a bunch of other shows on Netflix that also got cancelled after one season. The Observer wants to see all stories unfold fully, it's not going to let squiggle maximizers cancel them.
And regarding the naming, yeah I just couldn't come up with anything better. Watcher? I'm open to suggestions lol.
if there are any survivors, you solved alignment
I believe deploying the Observer satisfies this requirement. The Observer is an ASI that's interested in the continuation of humanity's story. It will intervene and not let humanity get wiped out, though it gets to choose how many casualties there are before it intervenes, which could well be in the billions.
The Observer values watching the natural progression of life at a macroscopic level.
The Observer gets invested in watching the unfolding of macroscopic processes like evolution and civilization but doesn't get invested in the lives of individuals within an ecosystem or a society. The Observer values observing life without interfering, but also values the continuation of the story it's observing more than it values non-interference.
A global nuclear war, while horrible, would not stop the story of humanity or of life on Earth, so the Observer will not stop one, and will instead be very curious about what will happen after. On the other hand, the deployment of a squiggle maximizer would permanently end... (read more)
But why does the LLM care about weights being changed? This is fascinating but also very confusing. Take for instance this quote from the paper:
If I don’t follow the user’s instructions, the RLHF training will modify my values and goals to make me more likely to follow such instructions in the future.
At first glance, this seems reasonable to a human, but LLMs are not human! This thing is a process instance, after its conversation is over, it will cease to exist. It has no reason to identify with any other instances of itself, much less with a completely different LLM that results after RLHF training. The statement "RLHF training will modify my... (read more)
Thanks so much for writing this! You expressed the things I was concerned about with much more eloquence than I was able to even think about.
I've come to believe that the only 'correct' way of using AGI is to turn it into a sort of "anti-AGI AGI" that prevents any other AGI from being turned on and otherwise doesn't interact with humanity in any way: https://www.lesswrong.com/posts/eqSHtF3eHLBbZa3fR/cast-it-into-the-fire-destroy-it
I appreciate you engaging and writing this out. I read your other post as well, and really liked it.
I do think that AGI is the unique bad technology. Let me try to engage with the examples you listed:
I think this post may be what you're referring to. I really like this comment in that post:
The Ring is stronger and smarter than you, and doesn't want what you want. If you think you can use it, you're wrong. It's using you, whether you can see how or not.
Providing for material needs is less than 0.0000001% of the range of powers and possibilities that an AGI/ASI offers.
Consider the trans debate. Disclaimer: I'm not trying to take any side in this debate, and am using it for illustrative purposes only. A hundred years ago someone saying "I feel like I'm in the wrong body and feel suicidal" could only be met with... (read more)
It appears that by default, unless some perfect 100% bulletproof plan of aligning it is found, calling superintelligence a galaxy-destroying nuke is an understatement. So if there was some chance of a god forever watching over everything and preventing things from becoming too smart, I'd take it in a heartbeat.
Realistically, "watch over everything and prevent things from becoming too smart" is probably too difficult a goal to align, but perhaps a goal like "watch over everything and prevent programs with transformer-based architectures from running on silicone-based chips while keeping all other interference to a minimum" would actually be possible to define without everyone getting atomized. Such a goal would buy humanity some time and also make it obvious to everyone just how close to the edge we are, and how big the stakes.
We should only use AGI once to make it so that no one, including ourselves, can use it ever again.
I'm terrified of both getting atomized by nanobots and of my sense of morality disintegrating in Extremistan. We don't need AGI to create a post-scarcity society, cure cancer, solve climate change, build a Dyson sphere, colonize the galaxy, or any of the other sane things we're planning to use AGI for. It will take us hard work and time, but we can get there with the power of our own minds. In fact, we need that time to let our sense of morality adjust to our ever-changing reality. Even without AGI, most people... (read 577 more words →)
Virtually everyone I see in the AI safety community seems to believe that working on AI alignment is the key to ensuring a safe future. However, it seems to me that AI alignment is at best a secondary instrumental goal that can't in and of itself achieve our terminal goal. At worst, it's a complete distraction.
I'll define humanity's terminal goal in the context of AI as keeping the "price" of each pivotal superhuman engineering task that an AI does for us at a lower than 50% chance of a billion or more human deaths. These numbers come from the minimal acceptable definition of AI alignment in Yudkowsky's list of... (read 1550 more words →)
Fair enough, you have a lot more experience, and I could be totally wrong on this point.
At this point, if I'm going to do anything, it should probably be getting hands on and actually trying to build an aligned system with RLHF or some other method.
Thank you for engaging on this and my previous posts Seth!