zroe1's Shortform

Zephaniah Roe

This is a special post for quick takes by Zephaniah Roe. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I feel like people haven't fully internalized what the world would look like if computer security actually broke.

There are some varying opinions on this, but a window of time without real computer security seems plausible. I was recently speaking with a computer security professor who I deeply respect and he was literally like "I think we are fucked and I don't think there is anything we can do."

This is similar to how people believe there is a 20% chance of extinction via AI but don't really internalize "No really. You will die and your girlfriend too. And your dog. And ..." In the cybersecurity case, some people believe (me included) that you cannot just patch all the bugs before releasing the model^[1] but then don't internalize "No really. It would be chaos. You may not be able to get into your bank account. Industrial plants could be compromised. Power could go out for several days at a time. [...]"

^{^}
The current plan seems to be to let companies use the models to fix all the bugs a model can find before a public release of that model. But there are so many companies that would need to do this properly for this to work, and the more companies you release to, the more opportunities there are for some bad actor to get their hands on the model. There are also many systems running legacy code that is very hard to update. In some cases you may need to go to a physical location to properly update a machine. Even if it is in theory possible to fix all the bugs, it would be pretty hard and I expect humanity to drop the ball for a while.

I would also like to see people's ideas on these. I would take a different approach than looking at the most outlandish or radical possibilities but rather try to assess the most likely and then add some about some interesting tail risk cases.

Not sure if these fit either of my criteria but one though I had regarding the banking/financial area. Do the older generation who was more used to in person activities and keeping paper trails survive better than the younger generation that grew up with only line banking and has done everything digitally?

It is true p... (read more)

2Shankar Sivarajan3mo

Perhaps this comment of mine from last year, on a post about "pessimization," essentially your observation that this tends to go quite badly, might help: (link) [...]

4Jason Gross3mo

It is definitely in theory possible to fix all the bugs. It is even in theory possible to make a box that will autonomously [1] fix all the bugs. The box consists of a theorem statement saying "attackers cannot breach the system", a proof checker, a sufficiently-but-not-unboundedly good code & proof model, and the instruction to prove the theorem statement while minimizing behavioral change, finding and fixing all bugs required to do this along the way. I am working on this box. The biggest question in my mind is whether we can get the capabilities in the right order to get this box in time. 1. Provided the box has the access to update the software, we have an adequate model of the hardware, the attacker model is software only and relatively standard (e.g., attacker is purely remote and can send and receive network packets), and we have a weak proxy for attacker-controlled action (something that never happens under normal operation, that we expect any attacker could in theory do, e.g., wipe the hard drive, send secret files directly over the network) ↩︎

zroe1's Shortform

6

zroe1's Shortform

6