Good points. People tend to confuse value pluralism or relativism with open-mindedness.
Contrasting this post with techniques like Word2vec, which do map concepts into spatial dimensions. Every word is assigned a vector and associations are learned via backprop by predicting nearby text. This allows you to perform conceptual arithmetic like "Brother" - "Man" + "Woman", giving a result which is a vector very close to (in literal spatial terms) the vector for "Sister".
Not sure what the LW consensus is, but there's some evidence that Roko's basilisk is a red herring.
Fortunately, a proper understanding of subjunctive dependence tells us that an
optimally-behaving embedded agent doesn’t need to pretend that causation can
happen backward in time. Such a sovereign would not be in control of its source
code, and it can’t execute an updateless strategy if there was nothing there to
not-update on in the first place before that source code was written. So Roko’s
Basilisk is only an information hazard if FDT is poorly understood.
From the post Dissolving Confusion around Functional Decision Theory.
Something like AI Dungeon but not so niche, like for emails. People paying money to outsource a part of their creative intellect is what appears significant to me.
Indeed. I feel like most of the work is done in the definition itself, which is necessarily paradigmatic in this case.
You mention that you're surprised to have not seen "more vigorous commercialization of language models" recently beyond mere "novelty". Can you say more about what particular applications you had in mind? Also, do you consider AI companionship as useful or merely novel?
I expect the first killer app that goes mainstream will mark the PONR, i.e. the final test of whether the market prefers capabilities or safety.
I don't know anyone who claims that it'll be a linear or unified experience. Without continuity and communication across instances, I don't think of it as personal immortality in the simple sense, any more than I think about children or great works as immortality.
Doesn't this also still apply to normal succession of mental states, without branching? How does QM or MWI come into play here?
There is no way to be "sent" anywhere because there is no soul to be shuttled around in the first place. There is only a sequence of experiences that are correlated with each other. In the dead branch there will be no experience, but in the living branch there will be, so the "illusion" of continuity (which is really just correlation) will be allowed to continue there.
Full disclosure: I haven't the slightest clue how quantum mechanics works.
I like your writing, but I don't think this piece belongs on LW. Sadly I don't have a good argument for why I believe that, except for the article's politics-adjacentness (though you did try to distinguish conservatism from the political right). I hope that you keep writing on this subject, but I'd prefer to not see it on LW in particular.
Do you consider perspective something experiential or is it conceptual? If the former, is there a shared perspective of sentient life in some respects? E.g. "suffering feels bad".