It's not really a fair question because we all have different things to do with our lives than launch snack lines or restaurant carts, but still: If people have discovered such an amazing delicious novel taste, both new and better than ice cream for 1/3 of those who try it, where are the people betting that it would be an amazing commercial success if only somebody produced more of it and advertised it more broadly?

And tbh, I wish I'd been there to try the food myself, because my actual first reaction here is, "Well, this sure is not a popular treat in supermarkets, so my guess is that some of my legion of admiring followers are so dead set on proving me wrong that they proclaimed the superior taste to them of something that sure has not been a wider commercial success, and/or didn't like ice cream much in the first place."

I have the most disobedient cultists on the planet.

What about this is supposed to be an infohazard rather than just private info? It doesn't seem like either a cognitohazard, negatively-valued information (movie spoilers), or a socioinfohazard / exfohazard (each individual prefers to know themselves but prefers society not to know).

See Simon Lerner above on how dead the horse appears to be.

So far as I can tell, there are still a number of EAs out there who did not get the idea of "the stuff you do with gradient descent does not pin down the thing you want to teach the AI, because it's a large space and your dataset underspecifies that internal motivation" and who go, "Aha, but you have not considered that by TRAINING the AI we are providing a REASON for the AI to have the internal motivations I want! And have you also considered that gradient descent doesn't locate a RANDOM element of the space?"

I don't expect all that much that the primary proponents of this talk can be rescued, but maybe the people they propagandize can be rescued.

Then I now agree that you've identified a conflict of fact with what I said.

Thank you for taking the time to correct me and document your correction.  I hope I remember this and can avoid repeating this mistake in the future.

Fair enough, but it was done with Anthropic's heavy and active cooperation to provide facilities not usually available to outside researchers, unless I'm mistaken about that too?

The gap between Before and After is the gap between "you can observe your failures and learn from them" and "failure kills the observer".  Continuous motion between those points does not change the need to generalize across them.

It is amazing how much of an antimeme this is (to some audiences).  I do not know any way of saying this sentence that causes people to see the distributional shift I'm pointing to, rather than mapping it onto some completely other idea about hard takeoffs, or unipolarity, or whatever.

I accept your correction and Buck's as to these simple facts (was posting from mobile).