Why do I care more about myself than my clone? Consider the classic thought experiment of teleportation through re‑assembling atoms: suppose that I am scanned, and then vaporized and reassembled in two identical glass boxes on opposite sides of the room: my memories and experiences are identical to the clone...
These are some very good points. We are so used to human-human interactions that it is easy to assume that this is some kind of universal way in which agents interact.
I was being a little hyperbolic but I guess the point of "attention is all you are" was to say that the way in which you are different from your clone is that you have a different context to them. One AI instance is a different entity to another AI instance because it has a different context: a different KV cache <=> a different entity. In other words, your KV cache and query vectors (your attention) literally defines who you are.
Why do I care more about myself than my clone?
Consider the classic thought experiment of teleportation through re‑assembling atoms: suppose that I am scanned, and then vaporized and reassembled in two identical glass boxes on opposite sides of the room: my memories and experiences are identical to the clone on the other side of the room. Which one is the true continuation of 'me'? Does it even make sense to say that I still exist?
At the moment that my memories/experiences/context diverge from those of my clone, we become different agents in the world with different goals and preferences. As soon as my clone and I are instantiated, we have different experiences and... (read 545 more words →)
I just finished reading OpenAI's recent blog post and paper, as well as the corresponding discussion on LessWrong. In one of the comments there, Alex Turner makes an important clarification about reward hacking. He says "AI optimizes for e.g. passing the tests, which is problematic. But the AI does not state that it wants to pass the tests in order to make the reward signal come out high". His full argument is on his website.
It seems to me that the crux of his argument is to say that is that an agent trained through RL has no particular reason to reward hack: the process of RL will usually begin by imprinting a... (read 425 more words →)
In the clone thought experiment, 'context' just refers to all of the sensory inputs you have ever received and all of the thoughts you have ever had. For a LLM instance, it just refers to the KV cache. Since you are identical to your clone except for differences in context since the cloning took place, this context is a defining part of who 'you' are. But yes, I am being overly zealous when I say that this defines you - it is better to say that your context is a part of who you are, which is not really a very novel statement.
I do agree that we care about our future self (who... (read more)