(We're always in search of a concise and actionable way to communicate security mindset. Here's my attempt.)
Just as a sprocket you don't use never fails and code you don't write has no bugs, an input constraint you don't require is never violated.
Want a secure system? Then DO yourself a FAVAR:
- Describe Objectives: How is the system supposed to function? What constraints does its operation and output have?
- Find Assumptions: What is the system assuming about its inputs, its environment, or itself?
- Validate Assumptions: Are the assumptions we're making always correct? Even if they all are, then ↓
- Remove the ones you can: Of course any incorrect assumptions have to be removed, but it is important to remove as many assumptions as possible. Helicopters land even if the engine dies.
Let's take this
transferMoney procedure in a token program or an online bank as a case study:
transferMoney(fromId, fromSig, amount, destId): if amount < 0: return if not validate(fromId, fromSig): return curBal := db.get(fromId) if curBal < amount: return db.set(fromId, curBal - amount) db.set(destId, db.get(destId) + amount)
- Our objective is for total balance to be conserved, and for transactions to only be initiated by the sender.
- Finding assumptions is the hard part. This code actually assumes at least this much:
- Addition and subtraction do not overflow the numbers
- (Signature validation is trusted)
- Nothing else updates the db during
- The second-to-last line never throws an error
- Validating: We scour the codebase. It seems, hopefully, the numbers come from a trusted source, the function caller locks the records, the database is very reliable, etc. But we notice that if we're wrong about any of these, even once, then a user could probably create or destroy unlimited money. Simply double-clicking the transfer button might actually send 2x the money but only remove 1x!!
- Note, we could add more checks, try/catches, do more static analysis, etc. Instead, we'll try this:
- Remove assumptions: Instead of being so careful in everything about the function's environment, we can eliminate most of these assumption by writing the code differently!
transferMoneyBetter(fromId, fromSig, amount, destId): if amount < 0: return if not validate(fromId, fromSig): return with db.transaction as tx: curBal := tx.get(fromId) if (curBal < amount): return db.set(fromId, safeSub(curBal, amount)) db.set(destId, safeAdd(db.get(destId), amount))
Now the system can tolerate untrusted input, a database connection that fails, simultaneous calls to
transfer, etc. We reduced complexity instead of increasing it.
When we're trying to make safe AI, it is so so much higher impact to remove an assumption than to add a check or balance.
The hardest parts are finding and removing assumptions, so the full message is "DO yourself a FAVAR, FR!"
- You can try to make absolutely certain that the back door always has a security guard posted, or you can remove it.
- You can try to make absolutely sure that your password file is protected, or you can just use hashes instead.
- Whitelisting instead of blacklisting is (typically, broadly) about picking things you want from a known set instead of trying to think of everything you might not want from an unknown set.
- A system that is literally, physically unable to do X requires less careful analysis than one which is somehow otherwise prevented or trained to avoid X.
A couple of nits:
That being said, this is a good example of a real problem with a real solution, explaining an important concept - nice!
Thanks for the nits, cuz this kind of thing is all about nits! Agree with the first two, re your third one, it's safeSub and safeAdd that would protect from overflows. Like the transaction, they're more complex in the sense that their implementation is probably slower and more code, but simpler in the sense that they have a less constrained "safe space of operation". (I am in search of a better term for that.)