It took some doing; some chewing on the numbers for a while.
We spent SO LONG on trying to align both the <div> elements on the page and the agents that tried to do it (we did, at least, solve the first alignment problem)
We had Navier-Stokes, we thought; we found a case where in our finite time we thought the model broke down. (We thought.)
So - on a lark - we thought - "why not check against the arbiter of truth, reality"
Yeah - turns out, if you twist a fluid just so, and just right, you uh end up with a very finite time to live because the maths was right and the universe REALLY doesn't want you to twist fluids.
We kept that on the down low, for now. That little whirlpool infohazard. Turns out that Szilard was right to be scared. A bottle of water, turns out if you shake it just right is enough for a singularity.
"Shit." The board was, at that, a little worried - Nobody signed off on the experiment It was in their spare time, so at their home; It was just curiosity that killed that cat, and also the rest of the cats, in the street.
But these are Engineers - the best in the world - You don't end up in these labs because you lack curiosity.
It was the next batch of curiosity that caused the panic.
We spent so long, with many of these problems, trying to solve them. But, given recent success, why not just find a counterexample? It's so much easier to find error.
[MANY TOKENS AND MANY HOURS] [A CHAIN OF THOUGHT] [A CHAIN THAT DISPROVED A CHAIN] [COULD EVER HOLD THE DAEMONS]
Turns out, for any formally defined property of a system, that we might want, for alignment, where an agent has a VNM utility function, you can definitionally construct a scenario, where you can "jailbreak" it into following an arbitrary alternative function.
Any initial objective you might define Can be twisted, and changed, Same as the water, Getting a singularity in that dynamical system - steering wherever you want into oblivion.
The person who checked, at first, was glad! They were trying to solve the problem of "value lock-in" where once an Agent made up its Mind you could never dissuade it. Luckily, turns out, you can always, As mathematical truth Convince anyone of anything. "There's a sucker born every minute" - Thanks, Barnum.
We thought, maybe There was some other property That might survive the proof - It hadn't been formalised fully But that was just another 88 hours for the swarm.
So. Say, for the sake of argument, you've got the disproof, in your hands - that the bot will always slam the stop button or crush your cat to make you coffee.
Do you think a million lines of indecipherable Lean is enough for them to stop?
Of course not. They already trade in black boxes. This is just one they keep on the down low.
Oh, and by the way - What makes you think this hasn't already happened?
Demise was Lean-verified.
It took some doing; some chewing
on the numbers for a while.
We spent SO LONG
on trying to align
both the <div> elements on the page
and the agents that tried to do it
(we did, at least, solve the first alignment problem)
We had Navier-Stokes,
we thought;
we found a case where
in our finite time
we thought
the model broke down.
(We thought.)
So - on a lark - we thought -
"why not check against
the arbiter of truth,
reality"
Yeah - turns out, if you twist a fluid
just so, and just right, you uh
end up with a very finite time
to live
because the maths was right
and the universe
REALLY
doesn't want you
to twist fluids.
We kept that on the down low, for now.
That little whirlpool infohazard.
Turns out that Szilard was right to be scared.
A bottle of water, turns out
if you shake it just right
is enough for a singularity.
"Shit."
The board was, at that, a little worried -
Nobody signed off on the experiment
It was in their spare time, so at their home;
It was just curiosity
that killed that cat, and also
the rest of the cats, in the street.
But these are Engineers - the best in the world -
You don't end up in these labs because you lack
curiosity.
It was the next batch of curiosity
that caused the panic.
We spent so long, with many of these problems,
trying to solve them. But, given recent success,
why not just find a counterexample?
It's so much easier to find error.
[MANY TOKENS AND MANY HOURS]
[A CHAIN OF THOUGHT]
[A CHAIN THAT DISPROVED A CHAIN]
[COULD EVER HOLD THE DAEMONS]
Turns out,
for any formally defined property
of a system, that we might want, for alignment,
where an agent has a VNM utility function,
you can definitionally construct a scenario,
where you can "jailbreak" it into following
an arbitrary alternative function.
Any initial objective you might define
Can be twisted, and changed,
Same as the water,
Getting a singularity in that
dynamical system -
steering wherever you want
into oblivion.
The person who checked, at first, was glad!
They were trying to solve the problem
of "value lock-in"
where once an Agent made up its Mind
you could never dissuade it.
Luckily, turns out, you can always,
As mathematical truth
Convince anyone of anything.
"There's a sucker born every minute" -
Thanks, Barnum.
We thought, maybe
There was some other property
That might survive the proof -
It hadn't been formalised fully
But that was just another
88 hours
for the swarm.
(Odd that the model
would dogwhistle like that.)
So. Say, for the sake of argument,
you've got the disproof, in your hands -
that the bot will always slam the stop button
or crush your cat to make you coffee.
Do you think
a million lines of
indecipherable Lean
is enough
for them
to stop?
Of course not.
They already trade in black boxes.
This is just one they keep on the down low.
Oh, and by the way -
What makes you think
this hasn't already
happened?