Researcher at Forethought.
Previously Longview Philanthropy and Future of Humanity Institute.
I interview people on Hear This Idea and ForeCast. My writing lives at finmoorhouse.com/writing. I myself live in London.
There are some social reasons for writing and reading blogs.
One reason is that “a blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox”. I expect to continue to value finding new people who share my interests after AI starts writing better blog posts than me, which could be very soon. I'm less sure about whether this continues to be a good reason to write them, since I imagine blog posts will become a less credible signal of what I'm like.
Another property that makes me want to read a blog or blogger is the audience: I value that it's likely my peers will also have read what I'm reading, so I can discuss it. This gives the human bloggers some kind of first-mover advantage, because it might only be worth switching your attention to the AI bloggers if the rest of the audience coordinates to switch with you. Famous bloggers might then switch into more of a curation role.
To some extent I also intrinsically care about reading true autobiography (the same reason I might intrinsically care about watching stunts performed by real humans, rather than CGI or robots).
I think these are relatively minor factors, though, compared to the straightforward quality of reasoning and writing.
As Buck points out, Toby's estimate of P(AI doom) is closer to the 'mainstream' than MIRI's, and close enough that "so low" doesn't seem like a good description.
I can't really speak on behalf of others at FHI, of course, by I don't think there is some 'FHI consensus' that is markedly higher or lower than Toby's estimate.
Also, I just want to point out that Toby's 1/10 figure is not for human extinction, it is for existential catastrophe caused by AI, which includes scenarios which don't involve extinction (forms of 'lock-in'). Therefore his estimate for extinction caused by AI is lower than 1/10.
Yes, I'm almost certain it's too 'galaxy brained'! But does the case rely on entities outside our light cone? Aren't there many 'worlds' within our light cone? (I literally have no idea, you may be right, and someone who knows should intervene)
I'm more confident that this needn't relate to the literature on infinite ethics, since I don't think any of this relies on inifinities.
Thanks, this is useful.
There are some interesting and tangentially related comments in the discussion of this post (incidentally, the first time I've been 'ratioed' on LW).
Thanks, really appreciate it!
Was wondering the same thing — would it be possible to set others' answers as hidden by default on a post until the reader makes a prediction?
I interviewed Kent Berridge a while ago about this experiment and others. If folks are interested, I wrote something about it here, mostly trying to explain his work on addiction. You can listen to the audio on the same page.
Thanks for the comment.
In denying certain properties of consciousness, illusionists are typically also denying that basic moral/axiological intuitions need to be grounded in them.
Obviously illusionists deny the inferences you are drawing, i.e. that it's fine to kill people, that people don't exist, or that they have no grounds to avoid being punched in the face. For those points to be more forceful, you need to show that (for example) it can pretty much only be bad to kill people because of the kinds of deep, extra phenomenal consciousness stuff which illusionists deny. That is, you are saying "if illusionism then [patently crazy conclusion], not [patently crazy conclusion], ∴ not illusionism". But the illusionist just denies "if illusionism then [patently crazy conclusion]".
Of course, one reaction is "it's totally obvious that illusionism implies crazy conclusions, if you can't see that, then we're living on different planets".
One reason I think a non-realist or illusionist research agenda is not inherently timid and mediocre, is that it is trying to answer very hard but pretty well-scoped empirical questions (e.g. the meta-problem) which don't currently have good answers, and where hypotheses are falsifiable. And I think that's a hallmark of successful scientific agendas: at the very least, it'll either generate interesting and testable new explanations, or it will fail. Compare realist approaches, where it's less clear to me how to decisively rule out bad explanations (because it's often unclear what testable predictions they are supposed to make). So I worry realist framings lack the engine for proper, cumulative progress.