I broadly agree.
I broadly agree. I think it is sometimes challenging to find the major pieces on an issue, though rarely super challenging (usually, if I just read the first few pieces I find and search citations backwards and forwards, at some point I find myself running into the same pieces over and over).
Sorry about the table of contents! The LessWrong versions of my posts are auto-generated (the originals appear here).
I think your comments about variance could technically be cast in terms of diminishing marginal returns. If having zero (or negative) impact is "especially bad", this implies that going from zero to small positive impact is "more valuable" to you than going from small positive to large positive impact (assuming we have some meaningful units of impact we're using). UH's argument is that this shouldn't be the case.
The point about variance eroding returns is an interesting one and not addressed in the piece. I think the altruistic equivalent would be something like: "If humanity stakes all of its resources on something that doesn't work out, we get wiped out and don't get to see future opportunities; if humanity simply loses a large amount in such fashion, this diminishes its ability to try other things that might go well." But I think the relevant actor here is mostly/probably humanity, not an altruistic individual - humanity would indeed "erode its returns" by putting too high a percentage of its resources into particular things, but it's not clear that a similar dynamic applies for an altruistic individual (that is, it isn't really clear that one can "reinvest" the altruistic gains one realizes, or that a big enough failure to have impact wipes someone "out of the game" as an altruistic actor).
I agree with Peter's comment.
> An alternative image is that the scientific fruit-pickers are building a scaffold, so that more fruit is always within arm's reach. Unless we're arguing that the scaffold is nearing the top of the tree of knowledge, there's always low-hanging fruit to be picked. Somebody will pick it, and whether or not they become a legendary figure depends on factors other than just how juicy their apple turned out to be. The reaching-for-fruit action is always equally effortful, but the act of scaffold-building gets more efficient every year. In literal terms, previous scientific discoveries and capital investments permit achievements that would have been inaccessible to earlier researchers, and we're getting better at it all the time.
I don't agree with this, for reasons discussed here. I think that empirically, it seems to get harder over time (at least per capita) to produce acclaimed works. I agree that there are other factors in who ends up "legendary," but I think that's one of them.
> Consider also that either this heuristic was either false at one point (i.e. it used to be the right decision to go into science to achieve "greatness," but isn't anymore), or else the heuristic is itself wrong (because so many obvious candidates for "great innovator" were all in academic science and working in established fields with fairly clear career tracks). If it used to be true that going into academic science was the right move to achieve scientific greatness, but isn't anymore, then when and why did that stop being true? How do we know?
The heuristic is "to match the greats, don't follow in their footsteps." I think the most acclaimed scientists disproportionately followed this general heuristic - they disproportionately asked important/interesting questions that hadn't gotten much attention, rather than working on the kinds of things that had well-established and -understood traditions and could easily impress their acquaintances. For much of the history of science, this was consistent with doing traditional academic science (which wasn't yet particularly traditional); today, I think it is much less so.
> How would you justify your "quasi realist" position. You want future Holden to look back on you. Why? Should others hold this preference? What if I wanted past Parrhesia to respect future Parrhesia. Should I weigh this more than future Parrhesia respecting past Parrhesia? I don't think this is meta-ethically justified. Can you really say there is nothing objectively wrong with torturing a baby for sadistic pleasure?
I don't think we can justify every claim with reference to some "foundation." At some point we have to say something like "This is how it seems to me; maybe, having articulated it, it's how it seems to you too; if not, I guess we can agree to disagree." That's roughly what I'm doing with respect to a comment like "I'd like to do things that a future Holden distinguished primarily by having learned and reflected more would consider ethical."
Thanks, this is helpful! I wasn't aware of that usage of "moral quasi-realism."
Personally, I find the question of whether principles can be described as "true" unimportant, and don't have much of a take on it. My default take is that it's convenient to sometimes use "true" in this way, so I sometimes do, while being happy to taboo it anytime someone wants me to or I otherwise think it would be helpful to.
I think this is still not responsive to what I've been trying to say. Nowhere in this post or the one before have I claimed that today's society is better morally, overall, compared to the past. I have simply reached out for reader intuitions that particular, specific changes really are best thought of as "progress" - largely to make the point that "progress" is a coherent concept, distinct from the passage of time.
I also haven't said that I plan to defer to future Holden. I have instead asked: "What would a future Holden who has undertaken the sorts of activities I'd expect to lead to progress think?" (Not "What will future Holden think?")
My question to you would be: do you think the changing norms about homosexuality, or any other change you can point to, represent something appropriately referred to as "progress," with its positive connotation?
My claim is that some such changes (specifically including changing norms about homosexuality) do - not because today's norms are today's (today may be worse on other fronts, even worse overall), and not because there's anything inevitable about progress, but simply because they seem to me like "progress," by which I roughly (exclusively) mean that I endorse the change and am in the market for more changes like that to get ahead of.
Does that clarify at all? (And are there any changes in morality - historical or hypothetical - that you would consider "progress?")
With apologies for the belated response: I think greghb makes a lot of good points here, and I agree with him on most of the specific disagreements with Daniel. In particular:
Overall, I think that datasets/environments are plausible as a major blocker to transformative AI, and I think Bio Anchors would be a lot stronger if it had more to say about this.
I am sympathetic to Bio Anchors's bottom-line quantitative estimates despite this, though (and to be clear, I held all of these positions at the time Bio Anchors was drafted). It's not easy for me to explain all of where I'm coming from, but a few intuitions:
Hopefully that gives a sense of where I'm coming from. Overall, I think this is one of the most compelling objections to Bio Anchors; I find it stronger than the points Eliezer focuses on above (unless you are pretty determined to steel-man any argument along the lines of "Brains and AIs are different" into a specific argument about the most important difference.)