This kind of scenario seems pretty reasonable and likely, but I'm much more optimistic about it being morally valuable. Mostly because I expect "grabbiness" to happen sooner and by an AI that is morally valuable.
I'm not entirely sure on the etiquette of posting something here and to your own site, so if there's a better way to do this, please let me know. Any other feedback or criticism would also be appreciated.
This is an update in the doomier direction for me, but it may be beneficial if it gets governments to start securing biolabs before future AI even exists.
If it's a choice between the genie giving me "the solution to AI Alignment" and the genie doing nothing, I'd take the solution and then spend the rest of my life testing it.
If I can use my wish for anything, I'd wish for some form of story-breaker power that I could verify easily and which would give the genie less room to screw me over.
This is a very nice addition to the collection of doomer short stories.
1Yes, this would be a bad situation to end up in, but I think it's extremely unlikely.
This is pretty clever. It reminds me of GANs in a way, but much more advanced. I know that the Pokemon-playing AIs on Twitch all have a version of "Critique Claude", which is a post-deployment version of this in some sense. Integrating that earlier in the process could be very useful. I'm not so sure how much this contributes to advancing capabilities vs advancing safety though, but I hope we'll get some good results from it.
I'd partially agree. I routinely see normal women who are more attractive than Sydney Sweeney or Gal Gadot, but they are still massively outnumbered by the women who aren't.
Avoiding what you suggested is why private conversations are an advantage. I think you misunderstood the essay, unless I'm misunderstanding your response.
I'll do that next time if that's the way.