Suppose there is a useful formulation of the alignment problem that is mathematically unsolvable. Suppose that as a corollary, modifying your own mind while ensuring any non-trivial property of the resulting mind was also impossible. Would that prevent a new AI from trying to modify itself? Has this direction been...
Disclaimer: there may be major flaws in the way I use words. Corrections are welcome. Suppose I want to memorize all the software design patterns. I could use spaced repetition and create a new deck of flashcards. Each card would have the name of the pattern on one side and...
The information content inferred is roughly "arbitrary byte sequences are the most predictable but least useful" and that could be expressed shorter!
This maps to king power to me.
How does this correlate with vocabulary changes? If the common vocabulary expanded, is the information contents of sentences still decreasing?
The first and third kinds of power are a bit vague.
They overlap, but don't fully cover the "I have a gun and you don't" kind of power.
It's not dominance because the person without the gun will or will not get shot regardless of displays of dominance. At least as long as the situation isn't already about dominance before the gun comes into play.
It's also not a form of getting-things-done power because the gun is a potential that exists even if no one involved currently has any reason to shoot anyone.
Agents want to be liquid.
An agent created in a computer would be an exception to that?
it would make the problem more tractable
The problem of creating a strong AI and surviving, that is. We'd still get Hanson's billions of self-directed EMs.
Thanks!
It has been explored (multiple times even on this site), and doesn't avoid doom. It does close off some specific paths that might otherwise lead to doom, but not all or even most of them.
Do you have any specific posts in mind?
To be clear, I'm not suggesting that because of this possibility we can just hope that this is how it plays out and we will get lucky.
If we could find a hard limit like this, it seems like it would make the problem more tractable, however. It doesn't have to exist simply because we want it to exist. Searching for it still seems like a good idea.
There's a hundred problems to... (read more)
Suppose there is a useful formulation of the alignment problem that is mathematically unsolvable. Suppose that as a corollary, modifying your own mind while ensuring any non-trivial property of the resulting mind was also impossible.
Would that prevent a new AI from trying to modify itself?
Has this direction been explored before?
This should be mitigated by pools of mutual trust that naturally form whenever there's a loop in the trust graph.
When the number of layers grows, the only thing that really works is metrics that cannot be goodhearted. Whenever those metrics exist, money becomes a perfectly good expression of success.
It might work to completely prohibit more than one layer of middle management. Instead, when middle manager Bob wants more people, he and his boss Alice come up with a contract that can't be gamed too much. Alice spins out Bob's org subtree into a new organization, and then it becomes Bob's job to buy the service from the new org as necessary. Alice also publishes the contract, so that entrepreneurs can swoop in and offer a better/cheaper service.
Disclaimer: there may be major flaws in the way I use words. Corrections are welcome.
Suppose I want to memorize all the software design patterns.
I could use spaced repetition and create a new deck of flashcards. Each card would have the name of the pattern on one side and the definition on the other.
This would help me understand references to patterns without opening Wikipedia every time. This would probably help me recognize patterns by descriptions, as long as they're close enough to the definitions.
But this wouldn't help me recognize patterns just by looking at their implementations. I'd have to actively think about each pattern I remember and compare the definition and the code.
I... (read more)
A bad codebase usually has more code and more bugs of common varieties. This requires a higher ability to read a lot, with lower depth of understanding.