Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

(We're always in search of a concise and actionable way to communicate security mindset. Here's my attempt.)

Just as a sprocket you don't use never fails and code you don't write has no bugs, an input constraint you don't require is never violated.

Want a secure system? Then DO yourself a FAVAR:

  • Describe Objectives: How is the system supposed to function? What constraints does its operation and output have?
  • Find Assumptions: What is the system assuming about its inputs, its environment, or itself?
  • Validate Assumptions: Are the assumptions we're making always correct? Even if they all are, then ↓
  • Remove the ones you can: Of course any incorrect assumptions have to be removed, but it is important to remove as many assumptions as possible. Helicopters land even if the engine dies.

Let's take this transferMoney procedure in a token program or an online bank as a case study:

transferMoney(fromId, fromSig, amount, destId):
	if amount < 0: return
	if not validate(fromId, fromSig): return
	curBal := db.get(fromId)
	if curBal < amount: return
	db.set(fromId, curBal - amount)
	db.set(destId, db.get(destId) + amount)
  • Our objective is for total balance to be conserved, and for transactions to only be initiated by the sender.
  • Finding assumptions is the hard part. This code actually assumes at least this much:
    • Addition and subtraction do not overflow the numbers
    • (Signature validation is trusted)
    • Nothing else updates the db during transfer's execution
    • The second-to-last line never throws an error
  • Validating: We scour the codebase. It seems, hopefully, the numbers come from a trusted source, the function caller locks the records, the database is very reliable, etc. But we notice that if we're wrong about any of these, even once, then a user could probably create or destroy unlimited money. Simply double-clicking the transfer button might actually send 2x the money but only remove 1x!!
    • Note, we could add more checks, try/catches, do more static analysis, etc. Instead, we'll try this:
  • Remove assumptions: Instead of being so careful in everything about the function's environment, we can eliminate most of these assumption by writing the code differently!
transferMoneyBetter(fromId, fromSig, amount, destId):
	if amount < 0: return
	if not validate(fromId, fromSig): return
	with db.transaction as tx:
		curBal := tx.get(fromId)
		if (curBal < amount): return
		db.set(fromId, safeSub(curBal, amount))
		db.set(destId, safeAdd(db.get(destId), amount))

Now the system can tolerate untrusted input, a database connection that fails, simultaneous calls to transfer, etc. We reduced complexity instead of increasing it.

When we're trying to make safe AI, it is so so much higher impact to remove an assumption than to add a check or balance.

The hardest parts are finding and removing assumptions, so the full message is "DO yourself a FAVAR, FR!"

More examples

  • You can try to make absolutely certain that the back door always has a security guard posted, or you can remove it.
  • You can try to make absolutely sure that your password file is protected, or you can just use hashes instead.
  • Whitelisting instead of blacklisting is (typically, broadly) about picking things you want from a known set instead of  trying to think of everything you might not want from an unknown set.
  • A system that is literally, physically unable to do X requires less careful analysis than one which is somehow otherwise prevented or trained to avoid  X.

See also

New Comment
2 comments, sorted by Click to highlight new comments since: Today at 12:26 AM

A couple of nits:

  • the second-to-last line throwing an error can be fine, as long as it happens before the data is written to the database. More worrying would be if the last line caused an error, as that's more likely to corrupt the data
  • the transaction hides the complexity, rather than removes it - the transaction adds some potential failure modes, even if it's a net lowering of the total complexity
  • a transaction might not protect from overflows - it will depend a lot on the actual implementations

That being said, this is a good example of a real problem with a real solution, explaining an important concept - nice!

Thanks for the nits, cuz this kind of thing is all about nits! Agree with the first two, re your third one, it's safeSub and safeAdd that would protect from overflows. Like the transaction, they're more complex in the sense that their implementation is probably slower and more code, but simpler in the sense that they have a less constrained "safe space of operation". (I am in search of a better term for that.)