AI #11: In Search of a Moat

We tried talking about AI Alignment, and that’s also not going so great.

Eliezer defined AI alignment as "the overarching research topic of how to develop sufficiently advanced machine intelligences such that running them produces good outcomes in the real world."

If you try to use this definition, and other people use AI alignment to refer to things that they think are relevant to whether advanced AI produces good outcomes, you can't really object as a matter of linguistics. I have no sympathy and am annoyed at the complaining. You can see the last LW discussion of this here.

I think "intent alignment" is a more useful concept, and complaining about how intent alignment isn't the same as existential risk from AI reflects a misunderstanding of how language works and what its purpose is. I think MIRI people are basically doomed to be perpetually frustrated because they are trying to use language as a tool to shape discourse in a way that's just not feasible. (The whole point of the choice of "alignment" was to sound good and bring in a bunch of connotations that aren't the same as the proposed meaning of the term.)

[-]TAG2y*42

Alignment seems to mean something that's alternative to control because of the ordinary meanings of the words. If you stipulate that you are using "alignment" to mean something else, you are going to face a perpetual uphill battle. If you define a dog as a domestic pet that either barks or meows, expect people to be confused.

Example:

https://www.lesswrong.com/posts/ykccy6LXfmTjZcp6S/separating-the-control-problem-from-the-alignment-problem

[-][anonymous]2y103

Not all accelerationism is based on some version of ‘well, what else are we going to do, you’ve strangled everything else and someone please help me my ~~family~~ civilization is dying’ or ‘I literally can’t envision a positive future anymore, so why not roll the dice.’

@Zvi I have to ask. In what timeline, with solely human intelligence and weak AI, does this trend reverse itself?

I simply don't see it. Humans are too stupid to ever relax the regulatory ratchet because the argument for a regulation sounds more convincing than the argument to relax it. Especially when government/high status institutions get to argue for restrictions (that happen to empower and guarantee their own jobs) while the argument against it usually comes from those with lower status.

AI research is difficult to impede because of the race mechanics/easy reproducibility in separate regulatory regimes, so it can actually proceed. Building more housing or balancing the budget or researching treatments for aging at a serious level? Impossible.

I'm an accelerationist for 2 main reasons.

(1) I dropped out of medical school, but not before witnessing that it's considered standard practice to have no plan at all. Hospitals exist to collect reimbursements, and their practitioners, who were all supposedly trained on empirical science, never even try, hell they don't even research cryogenically freezing any of their patients. This is so incredibly stupid you can't imagine any solution other than basically being able to fire everyone - that if you had a powerful enough set of ASIs, you could start your own hospital and deliver medicine that actually worked.

(2) It's not at all clear that, with constructible levels of compute and actual available data, that a being "smarter than us" would be smarter by the kind of margins that some are assuming. There is logarithmically less utility on real problems with greater intelligence, and this agrees with all AI research data I am aware of. The "sudden capabilities jumps" seem to be illusions. https://www.lesswrong.com/posts/qpgkttrxkvGrH9BRr/superintelligence-is-not-omniscience and other posts show that there are ultimate limits even for a perfect cognitive system. What this means is that narrow superintelligences that designed to stay focused on the problems we want to solve - with various forms of myopia in their agent designs - may in fact be controllable, with any attempts they have to break out or manipulate us failing trivially because they did not receive sufficient training data to succeed nor do they have enough information about the (target computer or target person) to succeed.

(3) if you try to play this out a little, if the default case is every human is going to die, then future worlds are either "machine men" (AIs that have all of our writings and images and DNA in files they learned from and some kind of civilization) or "men machines" (humans won, they made aligned AI and didn't die but the optimization pressures of future society turns everyone into immortal cyborgs with less and less organic parts)

If you're dead the 2 outcomes are indistinguishable, and it's hard to see how they are really any different. Either outcome is "every human alive now is dead, the information that made us human exists".

So yes, the cruxes are : the default case is everyone is going to die. Doesn't matter your age, the medical-research establishment as practiced by humans today will not develop a treatment for aging before the death of a newborn child alive today. And compute, especially inference compute, is so scarce today that if we had ASI right now, it would take several decades, even with exponential growth, to build enough compute for ASIs to challenge humanity.

[-]Zvi2y42

I don't see the reason for this defeatism - not on housing where YIMBY is actively winning some battles and gaining strength, not on aging where there might not be as much research as we'd like but there's definitely research and it will improve over time. As for balancing the budget, we did it as recently as the 1990s and also it's not obvious why we need to care about that.

So basically on your (1) I'd say yes we agree there are upsides I don't see how that leads to enough to justify the risks, and (2) I disagree strongly with the premise but even if you are right we would still be dead slightly slower, as your (3) suggests.

If your opinion is, roughly, 'I don't care if humans continue to exist once I am dead' then that would be a crux, yes. If I didn't care about humans existing after my death, I would roll the dice too.

[-]Lichdar2y30

AI research is difficult to impede because of the race mechanics/easy reproducibility in separate regulatory regimes, so it can actually proceed.

This is almost impossibly unlikely to produce good outcomes; this is selecting for speed by its ability to avoid our current means of alignment.

[-]Qumeric2y1-5

And compute, especially inference compute, is so scarce today that if we had ASI right now, it would take several decades, even with exponential growth, to build enough compute for ASIs to challenge humanity.

Uhm, what? "Slow takeoff" means ~1 year... Your opinion is very unusual, you can't just state it without any justification.

[-]awg2y811

Am I missing something about Grimes that warrants them getting 4 mentions in this post? Why should I care what Grimes in particular has to say about any of this (as opposed to someone like Geoff Hinton on the other extreme)?

I mean, Snoop Dogg is an interesting mention because in some ways his experience is a yardstick for how far these concepts are penetrating into mainstream society. But Grimes? Why Grimes?

[-]Zvi2y50

I mean, usually not Grimes, in this case the people I monitor were talking about her, and she is doing some interesting things (e.g. the AI voice thing) and it happened to take place four times in a week. She's a person, actually trying and thinking things, in addition to seeking attention and... we don't have as many people as we'd like.

[-]awg2y10

Thank you for replying!

[-]the gears to ascension2y50

she's good at powerseeking and keeps showing up in important places I guess

[-]awg2y42

This is a good answer for why she keeps showing up elsewhere. I'm asking why here, in these updates? I suppose I'm asking @Zvi this question directly.

[-]simon2y87

I also continue to see people assume that AI will increase inequality, because AI is capital and They Took Our Jobs. I continue to think this is very non-obvious.

It's quite reasonable in the long term on the model that AGI will become ubiquitous while respecting property rights (not saying I agree with this model).

For some time, I agree that AGI output would be offset by the humans being valuable in human-comparative advantage jobs. So even though all the former office workers have to get jobs as janitors in the AGI server farms (or whatever) at least they get lots of AGI-assisted services to have a high effective living standard.

But in the long term, the limiting factor is natural resources which get bid up by the AGI firms. So it becomes cheaper in the long run to build and maintain a robot to do the AGI server farm janitorial duties than to supply food for a human to eat, and human survival using labour as the source of income becomes non-viable.

See also Robin Hanson's If Uploads Come First.

[-]Corm2y70

Here’s some major mundane utility, huge if true: Automatically analyze headlines for their implications for stock prices, get there first and earn a 500% return

I read the article about this and their assumptions are insane. Normally stock news happens after the market closes. The way returns on the next day are calculated is just (end of day price for the day)/(end of day on the previous day). It is importantly not (end of day)/(opening price) of the stock. This is why stocks are able to very frequently open up many percentage points due to news happening overnight.

So, all they did was say based on the news we will get long if chatGPT says good and get short if chatGPT says bad, and the way we will calculate our returns is just whatever the returns for the next day are. But remember returns for the next day are based on the close for the previous day. So, they are assuming that if news comes out at 6pm they are able to buy or sell the relevant stock at exactly the 4pm closing price. This is a fucking insane assumption. Not only are markets very thin in the extended 4pm-8pm hours and pre open 4am-9:30am hours. But responding to news during non core trading hours (any time not 9:30-4:00) is already an extremely standard thing that hedge funds do. So yes this would be true if there was some magical market maker that was willing to market make at unlimited size in either direction without doing any fading in response to news or someone trading against them during the normally very thin non core trading hours.

[-]Zvi2y64

Now we don't have to read it! Not all heroes wear capes.

[-]Rudi C2y65

The most disappointing part of such discussions are the people who mean well, who under normal circumstances have great heuristics in favor of distributed solutions and against making things worse, not understanding that this time is different.

I have two great problems with the new centralist-doomer view, and I’d appreciate it if someone tried to address them.

Assuming the basic tenants of this worldview, it’s still not clear what threshold should be used to cut off open science. The old fire alarm problem if you will. I find it unlikely that this threshold just happens to be now that big economical contributions is possible while no signs of real dangers have been observed. The alternative hypothesis of rent seeking, OTOH, fits the hysteria perfectly. (I believe that EY probably believes we should have stopped open progress years ago, but I find that ridiculous.)
What happens if this scenario actually succeeds? How will it not be guaranteed to be a totalitarian nightmare? Unlike AGI, our history is full of examples of centralization casting people in hell.

My current belief is that centralist-doomers simply prefer being alive in any capacity whatsoever to being dead, and they are also under the hope/delusion that they will be part of the minority having power in this brave new world.

[-]Templarrr2y62

Only takes ~6 months to turn a non-AI researcher into an AI researcher

Um-hm, and it only takes a week to learn a syntax of programming language. Which in no way makes you a software engineer. I guess this really depends on the definition of "AI researcher". If the bar is "can do anything at all" without any measure of quality or quantity - 6 months is more than enough.

[-]RomanS2y61

This isn’t about the writers getting what should be due to them. That’s never going to happen. This is about the writers getting enough to survive, so they can keep writing. It is about not driving away the talent, and giving that talent the time and ability to develop.

Hollywood movies and shows are mostly harmful for cognition, and one of main culprits are the writers (explained in detail in this post of mine).

I think it's likely that replacing Hollywood writers with some GPT-5 will be a net improvement in this regard, as it will reduce the input from the people of the "I'm an irreplaceable Hollywood writer, a shepherd of mankind" mindset.

Could also make the movie scripts much more creative and entertaining (which is not a high bar, given the average quality of the Hollywood writing).

[-]Thane Ruthenis2y62

I have been using ‘AI existential risk’ which sounds reasonably serious and seems hard to co-opt or misunderstand. I haven’t entirely given up on alignment, but yes that might become necessary soon, and so far we don’t have a good replacement. In some sense, any ‘good’ replacement will get stolen.

"AI Omnicide Risk" is snappier, even less ambiguous, and has a grudging Eliezer approval, so we can coordinate around that term if we want it to sound respectable.

[-]Templarrr2y52

Our new band is called Foom Fighters – what are some good song titles?

Continuing the joke on the Meta-level - GPT-4 actually produces decent suggestions for these :)

"Echoes of the Algorithm"
"Neural Network Nightmare"
"Silicon Consciousness"
"Dystopian Data"
"Machine's Monologue"
"Binary Betrayal"
"AI Apocalypse"
"Deep Learning Delirium"
"Quantum Quandary"
"The Turing Test Tragedy"
"Ghost in the Machine"
"Singularity's Sorrow"
"Code of Consequence"
"Decoding Destiny"
"The Firewall Fallacy"
"Synthetic Shadows"
"Robot's Remorse"
"The Matrix Mirage"
"Deceptive Digits"
"Cybernetic Chains"

[-]Mitchell_Porter2y42

when the AI starts doing a lot of the prompt-creation automatically

This sounds like taking humans out of the loop.

One could make a series of milestones, from "AIs are finicky, and subtle differences in wording can produce massive gains in quality of reply", to "AI generally figures out what you want and does it well", to "AI doesn't wait for input before acting".

[-]Templarrr2y32

Humans can remember a 10-digit phone number in working memory – AIs will be able to hold the entirety of Wikipedia in working memory

In the context of LLM working memory is not it's training dataset. Training dataset in condensed and pattern-ized form is long term memory. Working memory is its "context window", so 8k or 32k tokens right now. Which on one hand is much better than 10digit number, but on the other - this comparison grossly underestimates the amount of data person holds in their "working memory" without thinking too much about it. "Where am I, what am I doing, why am I doing this, who passed me right now, who is sitting behind me, what is the tools I have available at the moment..." None of this we put in actual words inside our head but we still hold all of them in our working memory.

[-]Thane Ruthenis2y30

Zvi, any chance you can assemble your AI-newsletter posts into a sequence? I'd like to link people to them, and it's hard while they're uncategorized.

[-]Zvi2y40

Would be happy to have this done automatically by the mods, but am hesitant to introduce another weekly task I'd have to do by hand.

[-][anonymous]2y30

Judea Pearl is also Turing-awarded, and because of contributions to AI.

[-]TAG2y32

This could have done with a definition of "moat". Here's one I found:-

The engineer’s claim ties back to Buffett’s idea that, like medieval castles, successful businesses must protect their cash cows with moats that keep competitors from making a credible challenge. Those moats can come in the form of a size advantage that helps reduce costs, patents that protect key technological innovations, a high cost to entry for competitors, or even brand recognition. The billionaire detailed his economic moat theory in a 1999 Fortune article:

“The key to investing is not assessing how much an industry is going to affect society, or how much it will grow, but rather determining the competitive advantage of any given company and, above all, the durability of that advantage. The products or services that have wide, sustainable moats around them are the ones that deliver rewards to investors.”

[-]mesaoptimizer2y30

Formatting error: "OK, I used to work for a robotics company, and I do think that one of the key obstacles for a hostile AI is moving atoms around. So let me propose some alarms!" should be quoted since it is not you (Zvi) writing that passage but the person you linked and are quoting.

[-]TAG2y20

If you start out with an already superintelligent system, you do not get to ‘align’ it, further modify its preferences, because it will not let you do that.

As usual , that starts about three steps in. You need to first show that alignment as opposed to control is the only route to safety, that the supertinteligent system necessarily has goals of its own, and that it wants to be goal stable.