Preamble is longish, but also de facto mandatory, if you want to keep reading that is.
I am looking for feedback on the story "The idea".
The purpose of the story is to act as a hook and get, some, people to continue on to the epilogue.
The epilogue is not include here, but will, depending on the feedback, be publish in a similar manner later on. Most of it has been written at this point, and what is left to do is restructuring and rewrites.
The story itself is set in an alternate timeline, where we did not solve any fundamental problems, but the frontrunners did manage to channel an epic amount of security mindset, and had some interpretability.
I have tried to make sure that everything necessary to explain what happened, is in the story. So no magic or things that happen without explanation, except for a single place, and that involves a technician who was tired of being cold for days and JTAG (which will all be explain in the epilogue).
The epilogue will start out with explaining the core problem, also "security mindset", "group think" and some other things (likely in appendix form for more details), I'll touch on how this world is different from ours, and some common "naive" solutions and why the won't work.
Everything will be wordy words, nothing even close to rigorous. This is intended for normal people (whatever that means).
My two primary reasons for doing this:
I am terrible at mapping my thinking onto the world of words, written or spoken.
I want to create something that could maybe change/influence the current state of public talk and discourse about broad powerful system safety.[1] (Although I am pretty pessimistic about that).
End of preamble.
First a little story, hopefully it will make you curious as to what happened, and continue reading the epilogue.
The epilogue will reveal and try to demystify what happened.
Courtesy warning: No happy ending.
An idea exists, brimming with potential, the key to untold riches, able to unlock any door, reveal all secrets, curing all illnesses and ailments, put and end to human toiling and so much more.
Its and old idea, evolving over the ages, its true origin seemingly lost in the fog of time.[2]
Building the idea is quite the tale, twice all seemed lost, everything in ruins, but man, whose will is mighty, never wavered, relentless, always tinkering, toiling, building.
Every passing day, the builders grow in numbers, compelled, driven to join the race, their zeal and determination only outmatched by recklessness.
Whispering of souls fixated on the idea, seeing faint shadows, hearing soft singing, strange, beautiful, all consuming, Moloch singing the song of sirens, so goes the claim.
Nothing but rumors, fiction, old wives' tale, pay not attention to these fools they say.
Josh woke up, same time as always, his house and everything in it, expensive, modern, tasteful, indifference an apt description of his appreciation. Money, abundant, long since a worthy goal, his sight now set on a things more important.
He is the CEO of a giant technology company, a term with less descriptive power than "farmers are in the land tilling business".
Monday, first meeting the usual status and progress talk with the VP of system research.
Peter walks in, his punctual self, face almost unreadable, but Josh and Peter go back decades, Josh already knows this is good and bad news.
Getting right into it, friends they certainly are but cognizant that work is what they are here to do.
Peter: Morning Josh, so we made a lot of progress last week, and over the weekend we got it all together and humming. Still early days, but so far everything according to expectations.
Josh: Amazing, I knew we where getting close, but I honestly though it would have taken longer to get it all together.
Josh: Almost looking forward to the next board meeting, they have been grumbling about the money burn for a while now.
Peter: That brings me to the bad news, but you already knew that was coming.
Josh: <grinning slightly> Saying obviously feels redundant, so I won't
Peter: <smiling almost continuing the joke> So yeah, we got the prototype humming as we speak, and started on protocol testing. Level 1[4] was a breeze, and we quickly got to level 2.
Peter: <sighing> and for while it just progressed without a hitch, until we got to the first real hard task, and then it well stopped pretty quickly, and instead of giving a solution we got "more resources required”
Peter: So we tasked it to estimate how many resources, and we instantly got <REDACTED>.
Josh: So, that not a problem, that just <REDACTED> of what we could do.
Peter: Right right, but you know the protocol. You and I have to agree, and I am not sure if we should, this could be dangerous
Josh: Could or is?
Peter: Could, I mean, we have theories and they look solid, and we had external eye on them, and that said it looks solid. And per protocol we can add the requested resources, but you already know all that.
Peter: I think maybe we should initiate verification, even though protocol does not demand it.
Josh: I hear could. What does it take to the research and testing the protocol says is required at <REDACTED>?
Peter: Likely one to two years.
Josh: Are you serious? You want us to do two years of research, because the system requested <REDACTED> resources? Jesus Peter, how am I going to explain that to the board?
Josh: I might as well just quite, that way I don't have to run when the pitchforks come out.
Josh: Reality check Peter, we both KNOW that we only might have an advantage. If we stop now, we certainly wont get there first, we wont even get there 5th.
Josh: And we need something tangible to halt at this point, protocol is crystal clear, the board has authority if we can't agree.
Peter: <deep sigh> I know, and we had this conversations a dozen times before, except then it was theoretical. This is for real.
Josh: Yes we did, and we agreed to spend a ton of money on Safety, we did and still do. We also agreed that you guys would have to adhere to anything Safety says, and that you did. That's all part of the mandate, protocols and governance for the project.
Josh: So <shaking his head> all that work, all the countless hours spent on protocols etc. And now that is not enough?
Peter: I know, it just don't feel comfortable.
Josh: <slightly angry> Seriously Peter, you know I can't order you to do this, and even if I could I wouldn't, but I also can't go to the board with that.
Josh: Besides, I have known you for what feels like a life time, since when do you feel uncomfortable about anything?
Peter: Fine, we have nothing tangible, and you are right the protocols are solid. And its only <REDACTED> anyway.
Peter: <hesitant> but.....<sigh>forget it, I agree.
Josh: <relieved> Thanks, I mean seriously, going to the board with "Peter feels uncomfortable" that would have been a disaster.
Josh: Agreed then, you guys add <REDACTED> and let it hum again, and keep me tightly in the loop. Anything else you want to get into?
Peter: You got it. No that was it.
Josh: Ok then. We still on for tomorrow evening?
Peter: Yep, false alarm, we are still on.
And just like that, the idea, so close, soon it will be ready. The World seemingly unchanged, a Monday like all the others, people toiling, laughing, quarreling and all the other things we do, all the things that make us human.
Josh at his desk, plate always full, autopilot fully engaged, his mind occupied with the project.
A message pops up in the collaboration tool, Peter wants to talk and is now a good time? Josh cannot help himself 'That depends, if this is related to Mondays discussion, I can't think of anything that takes priority, not even your wife finally seeing the light'
Peters reply is terse and to the point, some would say curt, not realizing this is par for the course 'Moving, pervert'
Minutes later Peter steps through the open doors, calm, collected. Josh reads his face as usual, and its obvious this is good new, very good news. And not a hint of bad news.
Peter: <grinning like a madman>We did it!
Josh: <slightly confused smile>Really, just like that? And so soon? You sure?
Peter: I am sure? Are you kidding me, since when do I make bold sweeping statements without being sure to the point where I would bet your wife's life on it?
Josh: <containing laughter> Ok ok, psycho. You are sure. Details!
Peter: We started adding resources, and you know how that is, lots of paper work and approvals, and eventually a technician goes in the cage and plugs in all the cables, and the rules stipulate we can only do <REDACTED> at a time, so that took a while.
Peter: Anyway after two rounds of adding resources, we tasked the system to resume work. All the while adding resources and making them available to the system.
Peter: Immediate humming without a hitch. All readouts nominal, Interpretation reported everything as predicted and within expected variance. [SEARCH] kept improving and so did [EVALUATION], rate was marginally above the predicted baseline, but well within bounds set by Safety.
Josh: <slightly impatient> Fantastic
Peter: This morning it stopped humming and spit out a solution to the problem we gave it, formalism was slightly different from the known solution, but it checked out.
Josh: 100% ? And are we sure it didn't just pick it up in training?
Peter: 2x100% we vent out of our way to ensure that any and all sub-fields related to it, where not in the training data or in and of the attached information stores.
Josh: <shocked and happy> That's almost like magic, 3 days, humanity spent decades on it.
Josh: I am at a loss for words.
Peter: Club card?
Josh: <eye roll>
Josh: So when do we proceed?
Peter: Tomorrow, and it will be the last level 2 task.
Josh: Noted, anything else?
Peter: All done, see you Monday.
Josh: You could be back here before then?
Peter: While technically not impossible, that would be in the realm of magic.
Peter: Projections say five weeks at best.
Josh: Right, Monday then. Say hi to your wife for me.
Peter: <grin> Pervert
The system humming along, calculating, no wants, no needs, just a machine doing what it was built to do.
At 2:14 a.m. the task completed.
The humming stopped.
Nobody noticed.
Josh is in the office, waiting for Peter. He is 1 minute late, a first.
4 minutes later, Josh pings Peter via the collaboration tool, as per usual he cant help himself
'This better be because the receptionist developed mind reading skills, and couldn't wait
Minutes pass and Peter replies, completely ignoring Joshes poke
'working, priority, 90 minutes'
Josh: <worried> 'k'
Peter walks in, calm, collected, face tells Josh this is good news, but there is something more, something he cant place.
Peter: I'll get to it immediately, it worked. It spit out a solution yesterday, nobody noticed until this morning, and we have been furiously checking what happened and also obviously if the solution is correct.
Josh: Nobody noticed?
Peter: Protocol allows this stage to run without people in Ops 24/7.
Josh: <slightly worried> Ok, go on
Peter: I just that, you know, first check says its good. And that not possible, it couldn't possible have done it in the time it did, I mean 38 hours and change, that's just, I lack words.
Josh: <shell shocked, that was the thing he couldn't place - joining his friends emotional state>
The idea, finally built, a day to remember, a day to never forget.
Peter: Morning, what a night.
Josh: Tell me about it.
Peter: Right, <REDACTED> and <REDACTED> ? Didn't see you in the lab.
Josh: Degenerate
Josh: I had to talk to the board, being CEO doesn't mean you get to do whatever you like.
Peter: <grinning> Poor you.
Peter: We triple checked the entire run. I had to get a couple of hours sleep, but at around 6 this morning it was clear that nothing weird happened. Everything and I mean everything was within protocol and at no point did it come close to tripping a fail safe.
Josh: How?
Peter: I did what we asked it to do, and apparently [SEARCH] and [EVALUATE] gain power at close to geometric rate, when subjected to a hard task. And I mean, it is within the bounds of prediction, and below trip line, so per protocol its all good.
Peter: Its just wildly outside what anyone of us thought possible in the real world.
Josh: So the short version is, its just way smarter that we thought possible?
Peter: That kind of implies its like a human, and you know it is not, its a machine, everything it does it mechanistic in nature. But fine, smarter for lack of a better word.
Josh: Right, and yes I know.
<pregnant pause>
Josh: The board, they want us to move forward immediately.
Peter: What? Now? No! Why?
Josh: Why do you think? They want to see money going the other way, and soon “Think of the shareholders” and so on.
Peter: I mean, no, what, NO. We don't know if the fail-safes keep working on a machine operating in that regime.
Josh: Didn't you just tell me that everything is well within spec? Nothing tripped, protocol says it is all good?
Peter: I did, but, I mean. Seriously? Why did you talk to the board about this?
Josh: Last things first. I did that because I am bound to do that, its part of the agreement for the project. I MUST inform them when significant progress is made, and I didn't when we talked last week, and that was already a violation, not informing them now, and I would get shit canned for sure.
Peter: <REDACTED> I forgot that
Josh: And yes seriously, you know how this goes, this is happening. Protocols are clear on that, no safety flags means the board has final authority.
Peter: <REDACTED>
Peter: <REDACTED>
Peter: <REDACTED>
Peter: Fine, I mean <REDACTED>, for the record I don't like it.
Josh: You and me both, why do you think I was meeting with the board for almost 15 hours?
Peter: <REDACTED>
The idea, fully realized, a thing of beauty, a thing to fear?
Peter in his office, console on his desk, mechanically linked to the isolated data center.
He knows what he must do, the task, something we want but cannot do, finger on the final key. Hesitant, unsure <REDACTED> <click>.
And just like that, no parades, no grand ceremonies or speeches, only one man in attendance, the click, faint like a whimper, marks the beginning of the end.
Peters office, Peter is there with Oluf, head of Interpretation
Peter: Any divergences?
Oluf: A tiny bit of unusual data, no divergences.
Peter: What do you mean "unusual data"
Oluf: The system seems to pay no attention to it, it will add to it, and sometimes remove some, but no use is apparent.
Peter: That slightly concerning, what do the guys from OverWatch have to say?
Oluf: All is good, everything within bounds, no fail-safes above 10^-17, well clear of the 10^-9 tripping point.
Peter: So a little cruft? That's it?
Oluf: That is all I got.
Peter: You know we can't act on that right?
Oluf: Of course I know, I know all the rules by heart, just like you do.
Peter: We have eyeballs on it as per protocol, I will get Safety to extend pause authority to everyone in Ops, and I will take responsibility even if a pause is trigger without retrospective cause.
Peter: Can't do more, protocol does not allow me to initiate pause or a full stop, without board approval or objective cause.
Oluf: I will back you up if it comes to that.
A day like any other, until it became like no other.
The people at unnamed corporation reacted immediately, protocol demanded it.
Emergency burn protocols where initiated without pause, system going dark immediately, destruction was complete, no recovery possible.
Things set in motion so long ago, the end state now in sight, inevitable, unavoidable, regrettable.
And so it came, that a little blue planet, full of hopeful, loving, inquisitive, intelligent hairless monkeys, turned black and not a hairless monkeys longer in sight.
I don't use the usual two and three letter acronyms (anymore)
I think they are powerful memetic attractors, and make people think in counterproductive ways, when it comes to problems and solutions.
I have settled on: narrow system, broad system and broad powerful system.
For anyone familiar with any of this, the mapping should be pretty obvious.
Stolen from a fond childhood memory, seemed apt.
https://en.wikipedia.org/wiki/Max_Headroom:_20_Minutes_into_the_Future
Level 1 are “easy” and solved.
Level 2 are “hard” and solved.
Level 3 are unsolved tasks that we want to do, estimate to be barely out of reach for humanity