This open letter takes the form of annotations to Spearfield Teaching Hospital’s morbidity and mortality timeline for the incident on November 2, now widely known following the leak of confidential documents by some of my former colleagues. The public will be less familiar with certain crucial details which, in my view, exonerate Dr. Roe, myself, and Spearfield’s Department of Surgical Robotics, and provide what I believe to be an important lesson on the future of Machine Learning in healthcare and in general.

__________

02:00 – DARIUS monitoring shift II ends, shift III begins. Monitoring Room handover from Dr. [name redacted] to Dr. Roe. Dr. [name redacted] leaves. Dr. Roe begins monitoring shift.

To get a sense of Diagnostic Autonomous Robotic Intervention & Unsupervised Surgery’s relationship with – and isolation from – the outside world, it’s helpful first to understand the nested physical configuration of the Department of Surgical Robotics (DSR) within the Spearfield complex, and DARIUS itself at the heart of the DSR. 

The DSR comprises almost all of the northern arm of Spearfield’s cruciform campus, with only shared heating, ventilation and maintenance plant between the Department and the north employee-only entrance. To soothe the fearful, febrile minds of our institution’s administrators, DARIUS’s ambit is circumscribed by electromagnetically opaque, lead-rich walls, floors and ceilings that make it an island entirely enveloped by DSR property. 

It is not formally air-gapped – a single, low-bandwidth wire crosses the barrier to reach a dumb terminal in the Monitoring Room just outside, and there is a physical portal to admit and discharge patients and let technicians pass – but it has no access to the Internet, nor to Spearfield’s internal network, nor to any other network, and its miserable candle of an umbilical terminates well inside the DSR’s boundaries. 

Powerline scramblers even strip its electricity supply of all potentially data-carrying modulation. It is exquisitely firewalled; it is deaf and mute.

And it did not malfunction.

02:21 – Patient B (1-month survival likelihood 58%) moved from Oncology to AER after condition deteriorates.

Nurses will have brought B. (38, early-stage 4 liver cancer metastasised to the pancreas) to DARIUS’s portal, having received instructions from their superior to do so. Spearfield’s oncology department is large but busy, and especially during shift III when staffing is at its lowest, DARIUS routinely practices overflow emergency care in its Autonomous Emergency Room.

02:28 – DARIUS confirms poor vital signs of Patient B, administers emergency care which stabilises patient’s condition, determines hypercalcaemia of malignancy as cause of deterioration and adjusts Patient B’s 1-month survival likelihood to 53%.

02:33 – Donor admitted via AER in critical condition. DARIUS attaches EEG and ECG.

A 27-year-old with severe head trauma following a motorcycle accident.

02:35 – DARIUS measures Donor’s EEG to be macroscopically flat under 5µV/mm sensitivity, declares Donor brain dead, performs organ scan, finds pancreas, left eye, kidneys, left lung and heart viable, initiates surgery prep for Donor, transfers Patient A (1-month survival likelihood: 61.5%) from AW1 to OR4. 

Patient A., a 49-year-old admitted for pancreas transplant due to Type 1 diabetes and severe hypoglycaemic seizures not responding to insulin, had been awaiting a donor pancreas in Autonomous Ward 1 for two days. DARIUS was within its remit to move A. to Operating Room 4 without seeking human approval, on receipt of a viable donor organ.

I wish to digress briefly on the survival likelihoods in the timeline, all computed (in an eyeblink) by DARIUS. Similar estimates have been a staple of modern medicine since the field’s nascence; the novelty is that DARIUS’s are surpassingly accurate. Its core inference model has over six trillion trainable parameters and is abetted by forward-pass activation vectors to increase accuracy even further. After training, DARIUS’s survival estimates for an unseen 10,000-patient test cohort were accurate to within 0.12%.

(An uncomfortable but convincing confirmation of this astonishing performance is given by last year’s actuarial scandal, wherein three former DSR interns were jailed for attempting to induce relatives of patients with high mortality likelihoods to commit life insurance fraud.)

02:37 – Patient A’s vitals nominal after anaesthetic administration. 

There’s more to say about survival and probability. Popular press coverage of the incident, as well as displaying an almost arrogant inability to grasp the distinction between Bayesian and frequentist inference, regularly cited survival likelihoods without a time window. Without a time window, survival chances are meaningless. A healthy newborn has the same asymptotic chance of surviving as a nonagenarian chain-smoker: zero. It was frustrating, then, to hear inexpert journalists comparing DARIUS’s 1-month figures to human doctors’ (far less accurate) 6-month figures and finding the former wanting. 

02:38 – Patient A arrives at autonomous operating room OR4. Surgery begins.

Granted, a shift in perspective is necessary when applying statistical survival rates to a single patient. What does it mean for an individual to have a 61.5% chance of lasting a month? In thirty days’ time they will either be alive or be dead – where does the percentage apply? I’ll sidestep the philosophical wrinkles here and focus on what matters to the defence of Dr. Roe, myself and my department: the likelihoods obtain when choosing the best course of action based on priors and evidence. At a casino, all else being equal, do you choose to play a game with an 80% chance of winning or one with a 20% chance? The former, of course. And if you play the former and happen to lose, do you quail in superstitious fear, lose faith in the probabilistic information you have, and rashly switch to the other game? Of course not. As a dutiful Bayesian, you acknowledge that occasionally losing is compatible with your priors, you update them with this new data point, and you continue to bet on what is still  – and always will be – the sounder game.

02:47 – Dr. Roe leaves the monitoring room, failing to switch DARIUS to Minimal Intervention Mode.

Granted, Dr. Roe was technically at fault here. But particulars absent from the timeline are important. The worse for a refrigerated pasta e fagioli mistakenly judged extant, Roe experienced a sudden and pressing need for bathroom facilities. And in gastrointestinal panic a crucial step was missed. 

Minimal Intervention Mode is a universal state of DARIUS’s operating system that prevents it from making decisions with the slightest uncertainty of outcome. It can perform internal housekeeping like backing up checkpoints, self-testing and data compression, but is forbidden from taking physical action other than emergency care for patients at immediate risk of death. It is protocol for DARIUS to be in MIM whenever the Monitoring Room is unattended. (Another reactionary pearl visited upon us by a craven administration as a sop to a timorous public.)

We’ll revisit this, but for now the following should be noted: DARIUS is perfectly capable of automatically switching itself to MIM. Instead, focused as ever on PR perception over logic, management foisted that privilege on the fallible human-in-the-loop. The fearsome machine could not be trusted even to hobble itself.

02:48 – Patient A experiences seizure during surgery. DARIUS administers glucagon. 

02:50 – Patient A still in seizure, life signs deteriorating. DARIUS administers glucose 20% intravenous infusion, Patient A stabilises.

02:52 – DARIUS adjusts Patient A’s 1-month survival likelihood to 31% after transplant. Updated risk-benefit analysis requires that it seek outside approval for an altered course of action. DARIUS sends urgent request to monitoring room.

The editor of the timeline saw fit to salt away some crucial figures beneath the anodyne phrase ‘updated risk-benefit analysis’ – perhaps for fear of presenting the disjunction in the kind of plain, irrefutable, quantitative terms that the lay public tend to deplore. Tellingly omitted were Patient B.’s conditional survival likelihoods, of necessity also included in DARIUS’s risk-benefit computation: 53% (steady since first evaluated at 02:28) without multi-organ transplant; 74% with multi-organ transplant.

(Even when not in MIM, DARIUS must request human approval before taking actions that tilt the actuarial balance.)

02:54 – Receiving no response, DARIUS sends a second request.

Dr. Roe, poor soul, was still indisposed. 

02:57 – Receiving no response to the second request, DARIUS begins implementing its altered course of action.

I will say it again, though each repetition sounds more like bad science-fiction: the system did not malfunction. Its logs record in meticulous (and human-readable) detail the deductive reasoning on which it based its actions. DARIUS was fully aware (I know this is a loaded term and I do not at this time wish to imply subjective consciousness on the part of the model. I am merely stating that its internal state included symbols for all referents in its reckoning), fully aware of the consequences of its deliberation. 

DARIUS correctly made two attempts to obtain human authorisation through the only means available. Its LLM, symbolic AI layer and RL agent would have made short work of leveraging Internet communication channels, had it not been gagged and bound by technophobic decree: emergency services, telcos, social networks; even a text or email would have done the job.

Instead, having no voice and hearing only silence in response to its drastic whispers, it chose to do what would maximise the likelihood of best outcome. 

Can it even be called choosing? If ‘2’, ‘4’, ‘+’ and ‘=’ mean what we take them to mean, 2 + 2 = 4 can but follow. It is the inevitable consequence of perfect inference.

02:59 – DARIUS unstitches the partially-anastomised donor pancreas from patient A.

03:02 – DARIUS begins excising half of Patient A’s healthy liver and initiates prep and transfer of Patient B from AER1 to OR2. 

03:03 – Dr. Roe returns to monitoring room and attempts to cancel the new course of action. DARIUS temporarily enters Minimal Intervention Mode and requests confirmation of the cancellation order, listing risk likelihoods; Dr. Roe calls Chief of Autonomous Medicine Dr. Schnauss and explains the situation.

Earlier blunders notwithstanding, Dr. Roe was now acting with full competence. As was our machine, which was well within distribution to request confirmation of the cancellation order before following it. Roe had the presence of mind not to confirm in fright, but to read the summary data DARIUS had sent to the terminal. Not just the powerful argument of stark differences in expected outcome between the best and second-best alternatives; also the short interval before inactivity would bring death to both patients if the system continued idling in MIM for want of further instruction. Roe followed protocol by calling me.

(It should be obvious even to a lay reader that the balance of survival probabilities at this time – with the donor pancreas back in its transit ice bath and unfortunate A.’s liver part-excised – was even more skewed in B.’s favour than it had been eleven minutes earlier, when DARIUS judged it definitive.)

03:07 – Dr. Schnauss instructs Dr. Roe to let the autonomous course of action proceed. Dr. Roe does so.

To those who have not had it, it is hard to describe the subjective experience of trying to think analytically while lives and livelihoods hang in the balance; the ticking of each consequential second is like cannonfire in the eardrums. And yet I stand by my decisions. Did I opt to trust DARIUS implicitly in that moment, delaying vindication until I could pore over its logs ex post facto? Of course. The system operates unimaginably faster than the human mind. If I had insisted on verifying its logic before allowing it to proceed, both patients would have been in the morgue. Instead I put my faith in DARIUS doing exactly what we had designed it to do: thinking and acting more quickly, more calmly and more accurately than its designers.

03:09 – DARIUS transfers donor pancreas and Patient A’s partial liver from OR4 to OR2. Patient B is now prepped and ready for surgery in OR2.

03:12 – DARIUS transplants donor pancreas and Patient A’s partial liver into Patient B. Vital signs stable throughout. 

03:15 – Patient A undergoes hypoglycaemic seizure in OR4. DARIUS administers glucagon. 

Standard emergency treatment.

03:18 – Patient A still in crisis, life signs deteriorating. DARIUS administers glucose 20% intravenous infusion.

See previous entry.

03:20 – Patient B regains consciousness in OR2. Vital signs good.

It will surprise few readers that B. has been vocal in defending me, Dr. Roe and the Department, privately and publicly. We continue to receive notes of support from B.’s extended family, friends and colleagues. But I don’t wish to cite them here, or even submit them as evidence in our defence. My entire argument rests on DARIUS’s ability to make far-reaching and fateful decisions without bias, prejudice or preference. I follow its example here.

03:21 – Patient A worsens. DARIUS administers glucose 50% intravenous infusion.

See entry for 03:15.

03:24 – Patient A experiences catastrophic extravasation injury, enters ventricular fibrillation. DARIUS attempts to resuscitate via mechanical CPR. No response.

Antisymmetrically, no surprises regarding A.’s supporters’ opinions of the DSR or myself either. I am a “messianic narcissist who sees patients as research toys to be narcotised, perforated and discarded in the name of some automated anti-utopia”. I am an “amoral, unfeeling monster […] whose contraption should have never been given the power to decide who lives or dies”. Or most absurdly of all, I would, if I had my way, “preside over a mechanised human abattoir, feeding flesh to steel”.

I will let pass the melodramatic turn of phrase and cartoonish depiction of our hardware. I will resist the temptation to say again that the optimal course of action is a fact, not a decision. I truly feel for A.’s loved ones (there is barely a kilogram of steel in the entire – no, no, let it pass). I have devoted my entire professional life to healing. It is because I care that I stand by DARIUS.

(I refer tiresome amateur litigators, who look askance at DARIUS’s actions upon unconscious and therefore supposedly unconsenting patients, to pages 81 and 83 of the standard DSR waiver signed by all patients, and to pages 111-115 of Appendix 7 of Form LL1-V8 for patients with 1-month survival likelihoods below 70%, signed by A. on October 31 and by B. on November 1.)

03:26 – DARIUS attempts to resuscitate Patient A via defibrillation. No response.

As I write, DARIUS has been mothballed for eleven weeks, having been disconnected immediately after B. was transferred outside the DSR on November 3. Spearfield Teaching Hospital’s capacity, income and patient survival rate have all fallen in that time. The corridors of state-of-the-art 5-nanometer processing units in the DSR’s subterranean data centre, latent with potential to diagnose, to devise, to make better, lie dormant. No longer benefiting from its waste-heat recovery system, the campus must now pay for warmth.

All because of a single incident whose optics were bad. Not an accident, not a failure – just the wrong kind of success, at the wrong time. 

Every year, millions die in traffic accidents, and nobody calls for cars to be abolished. But when a new kind of intelligence forms a plan to minimise harm and executes it perfectly, the public howls in righteous fury.

03:29 – Patient A pronounced dead by DARIUS.

What we want from our ethics in the abstract, for some large diffuse group, can find itself in strong opposition to what we want when that generality is instantiated in the here and now, when the players are ourselves or those we love.

The solution to this problem of moral renormalisation is self-evident: enforce the abstract commandment fairly and consistently. But this is a difficult, perhaps impossible, task for a flesh-and-blood human with insuppressible biases. 

For DARIUS it is no effort at all.

In the long view, we will see this as a kindness.

Dr. G. K. Schnauss
Former Director
Department of Surgical Robotics
Spearfield Teaching Hospital