Introduction In 2020, around the time when I graduated with a master’s degree in computer science, I had a conversation with Steve Omohundro where we discussed his Basic AI Drives paper (among other things). At that time, there existed concrete demonstrations of where AI alignment could go wrong, but they...
I have recently read some blog posts (this, this and this) about tool AGIs and why agent AGIs are more likely. The argument that convinced me to err on the side of agent AGIs was that they are more likely to have an economical advantage over tool AGIs, as explained...
Introduction As is discussed in posts such as this one, a good way to test your skills as a machine learning research engineer is to implement a Transformer from scratch in PyTorch. This is exactly what I did. Below I am sharing my experience while doing so. The code I...
In 2020, around the time when I graduated with a master’s degree in computer science, I had a conversation with Steve Omohundro where we discussed his Basic AI Drives paper (among other things). At that time, there existed concrete demonstrations of where AI alignment could go wrong, but they were mostly limited to reinforcement learning. Since then, I worked on projects in various areas machine learning, mainly computer vision and natural language processing. During this time, I didn’t really keep up-to-date with the latest AI safety developments during that time.
However, recent developments in the field of AI safety have shown me why AI safety (and in particular, AI alignment) is a concrete... (read 1027 more words →)
Is there any area of AI safety research which answers research questions related to agency and what it means in the context of AGI agents?
I understand what you're saying, but I don't see why a superintelligent agent would necessarily resist being shut down, even if it has my own agency.
I agree with you that, as a superintelligent agent, I know that shutting me down has the consequence of me not being able to achieve my goal. I know this rationally. But maybe I just don't care. What I mean is that rationality doesn't imply the "want". I may be anthropomorphising here, but I see a distinction between rationally concluding something and then actually having the desire to do something about it, even if I have the agency to do so.
I have recently read some blog posts (this, this and this) about tool AGIs and why agent AGIs are more likely. The argument that convinced me to err on the side of agent AGIs was that they are more likely to have an economical advantage over tool AGIs, as explained here.
I do not question the value of AI safety research; I think it's good to have people who are trying to make sure that an agent AGI is aligned with what we (as humans) actually want. However, I am not sure that agency necessarily implies self-preservation instinct.
I think you could have a superintelligent agent (which has its own agency in the sense... (read more)
As is discussed in posts such as this one, a good way to test your skills as a machine learning research engineer is to implement a Transformer from scratch in PyTorch. This is exactly what I did. Below I am sharing my experience while doing so.
The code I wrote can be found in this GitHub repository, although I don't recommend looking at it if you are going to attempt this project yourself.
Hello Matthew,
I'm Mislav, one of the team members that worked on this project. Thank you for your thoughtful comment.
Yes, you understood what we did correctly. We wanted to check whether human preferences are "learned by default" by comparing the performance of a human preference predictor trained just on the environment data and a human preference predictor trained on the RL agent's internal state.
As for your question related to environments, I agree with you. There are probably some environments (like the gridworld environment we used) where the human preference is too easy to learn. On other environments, the human preference is too hard to learn and then there's the golden middle.
One of our... (read more)