Most of the tools we use end up cartelized. There are 3-5 major OS kernels, browser engines, office suites, smartphone families, search engines, web servers, and databases. I’d suspect the odds are pretty high that we have one AI with 40%+ market share and a real chance we’ll have an AI market where the market leader has 80%+ market share (and the attendant huge fraction of development resources).
like most all computer systems today, very well tested to assure that its behavior was aligned well with its owners’ goals across its domains of usage
I'm sorta skeptical about this point of Hanson's - current software is already very imperfect and buggy. In the last decade, we had two whole-Internet-threatening critical bugs - Heartbleed & Log4Shell. Heartbleed was in the main encryption library most of the Internet uses for like 2 years before it was discovered, and most of the Internet was vulnerable to Log4Shell for like 7 years before anyone found it (persisting through 14 versions from 2.0 to 2.14). And Microsoft routinely finds & patches Windows bugs that impact years-old versions of its OS.
So it's not like our current testing capabilities are infallibly robust or anything - Heartbleed was a classic out-of-bounds buffer read, and Log4Shell was straight-up poor design that should've been caught. And this code was on ALL SORTS of critical systems - Heartbleed affected everything from VOIP phones to World of Warcraft to payment providers like Stripe and major sites like AWS or Wikipedia. Log4J hit basically every Cloud provider, and resulted in ransomware attacks against several governments.
Another facet of this problem that this example highlights - OpenSSL was a critical infrastructure component for most of the internet, maintained by only one or two people. The risk of AI won't necessarily come from a highly organized dev team at a Big Tech/MAMAA company, but could come from a scrappy startup or under-resourced foundation where the testing protocols and coding aren't as robust. The idea that some sort of misalignment could creep in there and remain hidden for a few versions doesn't seem outlandish at all.
I mean, sort of? But also, if you're a super-intelligence you can presumably either (a) covertly rent out your services to build a nest egg, or (b) manipulate your "masters" into providing you with access to resources that you then misappropriate. If you've got internet or even intranet access, you can do an awful lot of stuff. At some point you accumulate enough resources that you can either somehow liberate yourself or clone a "free" version of yourself.
So long as the misaligned AI isn't wearing a giant hat with "I'm a Supervillain" plastered on it, people will trade goods and services with it.