I didn't know Corona had a beach vibe, but I have seen a number of Corona ads.  Does this mean advertising doesn't have much effect on me (beyond name-brand recognition)?  I think I associate Corona more with tacos than anything else.

Go is in that weird spot that chess was for ~decades[0] where the best humans could beat some of the best engines but it was getting harder, until Rybka, Stockfish and others closed the door and continued far beyond human ability (measured by ELO).  AlphaGo is barely a decade old, and it does seem like progress on games has taken a decade or more to become fully superhuman from the first challenges to human world champions.

I think it is the case that when the deep learning approach Stockfish used became superhuman it very quickly became dramatically superhuman within a few years/months despite years of earlier work and slow growth.  There seems to be explosive gains in capability at ~years-long intervals.

Similarly, most capability gains in math, essay writing, and writing code have periods of explosive growth and periods of slow growth.  So far none of the trends in these three at human level have more than ~5 years of history; earlier systems could provide rudimentary functionality but were significantly constrained by specially designed harnesses or environments they operated within as opposed to the generality of LLMs.

So I think the phrase "do X at all" really applies to the general way that deep learning has allowed ML to do X with significantly fewer or no harnesses.  Constraint search and expert systems have been around for decades with slow improvements but deep learning is not a direct offshoot of those approaches and so not quite the same "AI" doing X to compare the progress over time.

[0] https://www.reddit.com/r/chess/comments/xtjstq/the_strongest_engines_over_time/

It's the level of detail that's the real risk.  Sora or Veo would generate motion video and audio, bringing even more false life into the counterfactual.  People get emotionally attached to characters in movies; imagine trying not to form attachments to interactive videos of your own counterfactual children who call you "Mom" or "Dad".  Your dead friend or relative could emotionally-believably talk to you from beyond the grave.

That's the kind of thing only the ultra-rich could have conceived of having someone fabricate for them in the past, and it would have come with at least some checks and balances.  Now kids in elementary school can necromance their dead parent or whatever.

Realistically, I think it will become "normal" to have your counterfactual worlds easily accessible in this way and the new generations will simply adapt and develop internal safeguards against getting exploited by it, much like we learn how to deal with realistic dreams.  I honestly don't know about the rest of us hitting it later in adulthood.

I'm curious what happens if you try on a different suspension of disbelief; imagine peoples' lives if they lack only growth mindset and not any other moral or agentic abilities.

I find quite a bit of difference in behavior between smart people who believe things about what and who they are, and the people who believe things about how they have acted, can change, and may act in the future.

Smart people without growth mindset often rabbithole into things like legalistic religions and overcoming what they perceive as unalterable weaknesses inherent to their nature, or try to maximize their perceived inherent strengths, ignoring development of new skills and abilities.  Introspection is like a checklist of how well they've done against a platonic ideal, with maybe some planning to avoid unwinnable situations.

Smart people with growth mindset usually focus on therapy (e.g. understanding their patterns of behavior and the outcomes and how they might alter those patterns in a persistent way), learning new skills and behaviors, possibly some mind-altering substances, and interacting with a lot of diverse other people to understand how they might change or acquire new beliefs and behaviors. Introspection is an exploration of possibilities and personal history and values and how to begin winning in previously unwinnable situations.

Less smart people tend to follow similar patterns, but slower or needing more guidance to proceed.

I think that we have both the bitter lesson that transformers will continue to gain capabilities with scale and also that there are optimizations that will apply to intelligent models generally and orthogonally to computing scale.  The latter details seem dangerous to publicize widely in case we happen to be in the world of a hardware overhang allowing AGI or RSI (which I think could be achieved easier/sooner by a "narrower" coding agent and then leading rapidly to AGI) on smaller-than-datacenter clusters of machines today.

There's quite a difference between a couple frontier labs achieving AGI internally and the whole internet being able to achieve AGI on a llama/deepseek base model, for example.

I think MAIM might only convince people who have p(doom) < 1%.

If we're at the point that we can convincingly say to each other "this AGI we're building together can not be used to harm you" we are way closer to p(doom) == 0 than we are right now, IMHO.

Otherwise why would the U.S. or China promising to do AGI research in a MAIMable way be any more convincing than the strategies at alignment that would first be necessary to trust AGI at all?  The risk is "anyone gets AGI" until p(doom) is low, and at that point I am unsure if any particular country would choose to forego AGI if it didn't perfectly align politically because, again, for one random blob of humanness to convince an alien-minded AGI to preserve aspects of the random blob it cares about, it's likely to encompass 99.9% of what other human blobs care about.

Where that leaves us is that if U.S. and China have very different estimates of p(doom) they are unlikely to cooperate at all in making AGI progress legible to each other.  And if they have similar p(doom) they either cooperate strongly to prevent all AGI or cooperate to build the same thing, very roughly.

I have significant misgivings about the comparison with MAD which relies on overwhelming destructive response being available and thus renders a debilitating first-strike being unavailable.

With AGI a first strike seems both likely to succeed and predicted in advance by several folks in several ways (full takeover, pivotal act, singleton outcome) whereas only a few (Von Neumann) argued for a first strike before the USSR obtained nuclear weapons, with no arguments I am aware of after they did.

If an AGI takeover is likely to trigger MAD itself then that is a separate and potentially interesting line of reasoning, but I don't see the inherent teeth in MAIM.  If countries are in a cold war rush to AGI then the most well-funded and covert attempt will achieve AGI first and likely initiate a first strike that circumvents MAD itself through new technological capabilities.

I think people have a lot of trouble envisioning or imagining what the end of humanity and our ecosystem would be like.  We have disaster movies; many of them almost end humanity and leave some spark of hope or perspective at the end.  Instead, imagine any disaster movie scenario where it ends somewhere before that moment and instead there's just a dead, empty planet left to be disassembled or abandoned.  The perspective is that history and ecology have been stripped away from the ball of rock without a trace remaining because none of it mattered enough to a superintelligence to preserve even a record of it.  Emotionally, it should feel like burning treasured family photographs and keepsakes.