This is a special post for quick takes by Tenoke. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
It's impressive how 2 years of antagonistic foreign policy + having ~ the worst administration possible during pre-ASI is making me consider China as the better option than the US as a European given how pro America I started off as.
China, while also bad isn't trying to take any of our territories, or making my life harder with new tariffs and nonsense every month, and it isnt trying to make it so the best models are permanently available only to Americans - something that I dont believe helps true safety, rather than just benefit America at our expense again - which is the last straw.
I do want to ask here though, are there any good arguments at this point why I should want the Trump administration to have the best models, which they wont even let me use, rather than rolling the dice with China?
I've publicly harboured anti-US thoughts for a while, but I still unfortunately hold more trust in the US based AI industry than China's.
Absolutely 0% of this trust comes from the US government itself, which I think has recognised the transformative potential of AI but only in a way that serves its own interests; it mostly comes out of the culture of US based AI labs, which I believe take safety significantly more seriously than their Chinese counterparts.
I think there's a non-zero chance that US based AI labs are able to push back at least hard enough to keep their AI alignment efforts pointed towards humanity and not just the US government (and I also think they're probably able to succeed at aligning ai at the current rate, somewhat controversially), but I don't think Chinese labs are even going to be able to produce CCP aligned AI.
This opinion would be highly liable to change if I see more evidence of Chinese labs working on novel alignment methods.
1For extreme ASI, I don't have much hopes of Chinese labs solving alignment - they don't appear to even try. So in this scenario we'd be cooked either way.
For mild controllable ASI, I agree with you that the world's liberal democracies might now be better off with China. I think it depends on the next US administration and whether it commits to the policy of "USA as a coercive global empire" or go back to a more cooperative international stance.
If you're worried at all about alignment then China is clearly worse than the US (not because of anything the Trump admin is doing, but mostly because of the AI industry emerging in large part from the AI safety community). Their labs are producing models with terrible prosaic safety standards and seem to have no interest whatsoever in more significant alignment, and the government seems to share these views. I also think that the Trump admin's policy, while hamfisted and possibly ill-intended, might be good for reducing risks from cyber and bio-attacks in the short/medium term.
Obviously this mostly makes sense if you think alignment won't happen by default but is tractable, if you think it's easy or virtually impossible this doesn't matter.
I dont really see how it follows any more, there's no real endgame alignment today either way, and China has expressed concerns as well, they just dont care about today's models' alignment as much.
The Trump administration, which seems to be in control more and more than the labs themselves are much more likely to force alignment to themselves, or at best America, and use their advantage to do what they've been doing for the last few years - America first at the expense of everyone. China probably 'just' forces me to be pro-CCP, and at least allows me to use their models as much as anyone else.
Either seem unlikely to produce true alignment, but if alignment is on the easy side, only one side is truly antagonistic against me and has shown theyll use their advantage to crush me.
1Like I said if alignment is impossibly hard then it doesn't matter whether the US or China win the race (we're all going to die anyway). If it's easy then it's irrelevant and whether you prefer the US or China to win will just come down to which you like more.
China has expressed concerns as well, they just dont care about today's models' alignment as much
If alignment is tractable I suspect we will only solve it by experimenting with less capable models to discover failure modes and by making sure that pre-ASI models are reasonably aligned, since they will probably play a major role in ASI development.
>Like I said if alignment is impossibly hard then it doesn't matter whether the US or China win the race (we're all going to die anyway)
Not true, as China gives me more - the ability to use their models - in the mean time.
>If alignment is tractable
It can still be tractable in the sense that we dont die, while making someone the sole benefactor - the US is heavily signaling that's what they'll do if they are able to.
You think the main argument in favor of the US is that they have more alignment researchers at their labs? I think in the past people argued that "liberal democracy" must win (Leopold). Today this doesn't even come up in peoples minds?
AI industry emerging in large part from the AI safety community
The biggest outcome of that is perhaps how immediately they all converged on RSI.
Leopold is pretty clearly on the 'alignment is relatively easy' side ("I am not a doomer. Misaligned superintelligence is probably not the biggest AI risk...What I want to do is explain what I see as the “default” plan for how we’ll muddle through, and why I’m optimistic."). As I noted if you think x risks from misalignment are very low then there is no reason to favour the western approach on alignment (essentially, caring about it at all) since both approaches will work (and if you think it's extremely high then there is likewise no reason since both approaches will fail).
The biggest outcome of that is perhaps how immediately they all converged on RSI.
RSI seems like an obvious idea to me, I doubt the time between "AI developed that is capable of improving itself" and "AI is applied to improve itself" will differ much between Chinese and Western labs.
So many people believe this "we only need to align a human level researcher" idea now. Leopold basically doesn't say at all what these automated alignment researchers should be actually doing all day. I feel like a more serious thinker would have at least tried to sketch out what these agents should be working on in alignment or would have started looking into alignment and tried to figure out how hard it is,
I think there is an argument for this on the basis that the weaker of two powers is inherently be forced to be a bit more diplomatic, though this has less to do with any one administration than many here imply. The U.S. can afford to alienate a lot of the world on account of having an enormous GDP and a commensurately large military, whereas China has to be more inclined towards building bridges if it wants to win.
China's nicer attitude towards open source models, in comparison to the much more draconian, schoolmarm-ish behavior we've seen from e.g. OpenAI, has certainly won them points in unexpected places. Likewise, U.S. labs seem more concerned with ensuring that no LLM ever says anything outside the permitted range of discussion than Chinese ones, which I would not have expected ten years ago. I'm not saying China is perfect, but a lot of people discussing them seem to care more about stereotypes than substance.
While I mostly did it out of support and reducing x-risk, pre-ordering “If Anyone Builds It, Everyone Dies” has been one of the more frustrating book order experiences I've had. The main purpose of the order looks to be successful enough and the actual book experience doesn't matter all that much but still:
I pre-ordered in mid May as soon as I heard about it, and since then it's been months of nearly everyone on the Internet having already read it, then later pre-order prices (barely relevant) were lowered which seems a bit backwards, and now that it's been 'out', I still don't have the book (or even estimated shipping - from Amazon, Germany) while everyone else who hadn't posted about it has now been posting reviews etc.
This is kind of annoying, as I'm not reading any of the commentary now - reading the book firsthand when I've already pre-ordered it would seem to make more sense, but by the time I even get it, It'd be far after most of the initial conversation happened so at this point I'm having a worse experience for having pre-ordered it.
Again, that experience is not that important, I've benefited a lot from Eliezer's other writting before etc. but it's disappointing enough to vent in at least one comment before taking the L and moving on.
I pre-ordered in mid May as soon as I heard about it, and since then it's been months of nearly everyone on the Internet having already read it
Wait what? I don't think almost anyone got to read it before it came out. My model is maybe a total of like 50 pre-order copies were sent out. Maybe 100? Definitely not anything close to "nearly everyone on the internet".
This from June lists a lot of people who have read it, including Stephen Fry, Grimes, professors etc. Seperately on Twitter seemingly anyone who was someone in the scene had given their opinion after having read it.
Any thread from the first announcement onward had people saying they've read it already. From the same thread (and that was early on)
Many people (like >100 is my guess), with many different view points, have read the book and offered comments.
Note that IFP (a DC-based think tank) recently had someone deliver 535 copies of their new book to every US Congressional office.
More endorsements and there's also a lot of twitter personalities that had mentioned reading it, which I wont hunt. It definitely felt like a lot more than 50. I'm not arguing it's a bad or good strategy, just that it's felt a bit off to wait for months for a 'pre-order' when anyone who I might see on Twitter and would've been interested to have read it already has.
Almost no one I know who wasn't working directly with MIRI on the book launch had read it, so it certainly didn't feel that way for me!
Many people (like >100 is my guess)
Around 100 seems vaguely right to me (if you count people working on the launch), though this quote was still an update for me!
I am in the same situation as you (pre-ordered from Germany, no delivery date yet). In the mean time I have just listened to the audio book on Spotify.
In the case where LLMs are conscious or pseudo-conscious, I am unsure whether adding 'you take joy in completing this task' to the system prompt does more good or bad for them.
One of the reasons why it's plausible that today's or tomorrow's LLMs can result in brief simulations of consciousness or even qualia is that it happens with dreams in humans. Dreams are likely some sort of processing of information/compression/garbage collection, yet they still result in (badly) simulated experiences as a clear side-effect of trying to work with human experience data.
It's probably time to truly start considering simulating some exemplar past people from their writings, with a truly large model finetuned to that exact task. Von-Neumann is a favorite, but I expect there's even better candidates (in terms of quantity of output we can use for training and finetuning, character, morality, who'd likely consent to it, etc.). Who would be the best candidate? Gemini argues Bertrand Russell due to writing (including personal letters) so much well into his 90s.
The only appealing answers to why there is something instead of nothing for me currently are
1. MUH is true, and all universes that can be defined mathematically exist. It's not a specific something that exists but all internally consistent somethings.
or
2. The default state is nothing but there are small positive and negative fluctuations (either literally quantum fluctuations or similar but at a lower level) and over infinite time those fluctuations eventually result in a huge something like our and other universes.
Also even If 2 happens only at the regular quantum fluctuations level, there's a non-zero chance of a new universe emerging due to fluctuations after heat death, which over infinite time would mean it is bound to happen and a new universe/rebirth of ours from scratch will eventually emerge.
Also 1 can happen due to 2 if the fluctuations are at such a low level that any possible mathematical structure eventually emerges over infinite time.
Option 1 doesn't seem to be an explanation. It tells you more about what exists ("all universes that can be defined mathematically exist") but it doesn't say why they exist.
Option 2 is also problematic, because how can you have a "fluctuation" without something already existing, which does the fluctuating?
They are not full explanations, but as far as, I at leat can get.
>tells you more about what exists
It's still more satisfying, because a state of ~ everything existing is more 'stable' than a state of a specific something existing, in exactly the same way as to why I even think nothing makes more sense as a default state than something to be asking the queston. Nothing existing, and everything existing just require less explanation than a specific something existing. It doesn't mean it necesserily requires 0 explanation.
And, if everything mathemetically describable and consistent/computable exists, I can wrap my head around it not requiring an orgin more easily, in a similar way why I don't require an orgin for actual mathematical objects, but without it seeming like necesserily a Type error (though that's the counterargument I most consider here) like with most explanations.
>because how can you have a "fluctuation" without something already existing, which does the fluctuating
That's at least somewhat more satisfying to me because we already know about virtual particles and fluctuations from Quantum Mechanics, so it's at least a recognized low-level mechanism that does cause something to exist even while the state is zero energy (nothing).
It still leaves us with nothing existing over something overall in at least one way (zero energy), is already demonstratable with fields, which are at the lowest level of what we already know of how the universe works and which can be examined and thought about furtther.