HoldenKarnofsky

Sequences

The Most Important Century

Wiki Contributions

Comments

I broadly agree. I think it is sometimes challenging to find the major pieces on an issue, though rarely super challenging (usually, if I just read the first few pieces I find and search citations backwards and forwards, at some point I find myself running into the same pieces over and over).


 

Sorry about the table of contents! The LessWrong versions of my posts are auto-generated (the originals appear here).

I think your comments about variance could technically be cast in terms of diminishing marginal returns. If having zero (or negative) impact is "especially bad", this implies that going from zero to small positive impact is "more valuable" to you than going from small positive to large positive impact (assuming we have some meaningful units of impact we're using). UH's argument is that this shouldn't be the case.

The point about variance eroding returns is an interesting one and not addressed in the piece. I think the altruistic equivalent would be something like: "If humanity stakes all of its resources on something that doesn't work out, we get wiped out and don't get to see future opportunities; if humanity simply loses a large amount in such fashion, this diminishes its ability to try other things that might go well." But I think the relevant actor here is mostly/probably humanity, not an altruistic individual - humanity would indeed "erode its returns" by putting too high a percentage of its resources into particular things, but it's not clear that a similar dynamic applies for an altruistic individual (that is, it isn't really clear that one can "reinvest" the altruistic gains one realizes, or that a big enough failure to have impact wipes someone "out of the game" as an altruistic actor).


 

> An alternative image is that the scientific fruit-pickers are building a scaffold, so that more fruit is always within arm's reach. Unless we're arguing that the scaffold is nearing the top of the tree of knowledge, there's always low-hanging fruit to be picked. Somebody will pick it, and whether or not they become a legendary figure depends on factors other than just how juicy their apple turned out to be. The reaching-for-fruit action is always equally effortful, but the act of scaffold-building gets more efficient every year. In literal terms, previous scientific discoveries and capital investments permit achievements that would have been inaccessible to earlier researchers, and we're getting better at it all the time.

I don't agree with this, for reasons discussed here. I think that empirically, it seems to get harder over time (at least per capita) to produce acclaimed works. I agree that there are other factors in who ends up "legendary," but I think that's one of them.

> Consider also that either this heuristic was either false at one point (i.e. it used to be the right decision to go into science to achieve "greatness," but isn't anymore), or else the heuristic is itself wrong (because so many obvious candidates for "great innovator" were all in academic science and working in established fields with fairly clear career tracks). If it used to be true that going into academic science was the right move to achieve scientific greatness, but isn't anymore, then when and why did that stop being true? How do we know? 

The heuristic is "to match the greats, don't follow in their footsteps." I think the most acclaimed scientists disproportionately followed this general heuristic - they disproportionately asked important/interesting questions that hadn't gotten much attention, rather than working on the kinds of things that had well-established and -understood traditions and could easily impress their acquaintances. For much of the history of science, this was consistent with doing traditional academic science (which wasn't yet particularly traditional); today, I think it is much less so.


 

> How would you justify your "quasi realist" position. You want future Holden to look back on you. Why? Should others hold this preference? What if I wanted past Parrhesia to respect future Parrhesia. Should I weigh this more than future Parrhesia respecting past Parrhesia? I don't think this is meta-ethically justified. Can you really say there is nothing objectively wrong with torturing a baby for sadistic pleasure?

I don't think we can justify every claim with reference to some "foundation." At some point we have to say something like "This is how it seems to me; maybe, having articulated it, it's how it seems to you too; if not, I guess we can agree to disagree." That's roughly what I'm doing with respect to a comment like "I'd like to do things that a future Holden distinguished primarily by having learned and reflected more would consider ethical."

Thanks, this is helpful! I wasn't aware of that usage of "moral quasi-realism."

Personally, I find the question of whether principles can be described as "true" unimportant, and don't have much of a take on it. My default take is that it's convenient to sometimes use "true" in this way, so I sometimes do, while being happy to taboo it anytime someone wants me to or I otherwise think it would be helpful to.

I think this is still not responsive to what I've been trying to say. Nowhere in this post or the one before have I claimed that today's society is better morally, overall, compared to the past. I have simply reached out for reader intuitions that particular, specific changes really are best thought of as "progress" - largely to make the point that "progress" is a coherent concept, distinct from the passage of time.

I also haven't said that I plan to defer to future Holden. I have instead asked: "What would a future Holden who has undertaken the sorts of activities I'd expect to lead to progress think?" (Not "What will future Holden think?")

My question to you would be: do you think the changing norms about homosexuality, or any other change you can point to, represent something appropriately referred to as "progress," with its positive connotation?

My claim is that some such changes (specifically including changing norms about homosexuality) do - not because today's norms are today's (today may be worse on other fronts, even worse overall), and not because there's anything inevitable about progress, but simply because they seem to me like "progress," by which I roughly (exclusively) mean that I endorse the change and am in the market for more changes like that to get ahead of.

Does that clarify at all? (And are there any changes in morality - historical or hypothetical - that you would consider "progress?")

With apologies for the belated response: I think greghb makes a lot of good points here, and I agree with him on most of the specific disagreements with Daniel. In particular:

  • I agree that "Bio Anchors doesn't presume we have a brain, it presumes we have transformers. And transformers don't know what to do with a lifetime of experience, at least nowhere near as well as an infant brain does." My guess is that we should not expect human-like sample efficiency from a simple randomly initialized network; instead, we should expect to extensively train a network to the point where it can do this human-like learning. (That said, this is far from obvious, and some AI scientists take the opposite point of view.)
  • I'm not super sympathetic to Daniel's implied position that there are lots of possible transformative tasks and we "only need one" of them. I think there's something to this (in particular, we don't need to replicate everything humans can do), but I think once we start claiming that there are 5+ independent tasks such that automating them would be transformative, we have to ask ourselves why transformative events are as historically rare as they are. (More at my discussion of persuasion on another thread.)

Overall, I think that datasets/environments are plausible as a major blocker to transformative AI, and I think Bio Anchors would be a lot stronger if it had more to say about this. 

I am sympathetic to Bio Anchors's bottom-line quantitative estimates despite this, though (and to be clear, I held all of these positions at the time Bio Anchors was drafted). It's not easy for me to explain all of where I'm coming from, but a few intuitions:

  • We're still in a regime where compute is an important bottleneck to AI development, and funding and interest are going up. If we get into a regime where compute is plentiful and data/environments are the big blocker, I expect efforts to become heavily focused there.
  • Several decades is just a very long time. (This relates to the overall burden of proof on arguments like these, particularly the fact that this century is likely to see most of the effort that has gone into transformative AI development to date.)
  • Combining the first two points leaves me guessing that "if there's a not-prohibitively-difficult way to do this, people are going to find it on the time frames indicated." And I think there probably is:
    • The Internet contains a truly massive amount of information at this point about many different dimensions of the human world. I expect this information source to keep growing, especially as AI advances and interacts more productively and richly with humans, and as AI can potentially be used as an increasingly large part of the process of finding data, cleaning it, etc.
    • AI developers will also - especially as funding and interest grow - have the ability to collect data by (a) monitoring researchers, contractors, volunteers, etc.; (b) designing products with data collection in mind (e.g., Assistant and Siri). 
    • The above two points seem especially strong to me when considering that automating science and engineering might be sufficient for transformative AI - these seem particularly conducive to learning from digitally captured information.
    • On a totally separate note, it seems to me that fairly simple ingredients have made the historical human "environment" sufficiently sophisticated to train transformative capabilities. It seems to me that most of what's "interesting and challenging" about our environment comes from competing with each other, and I'd guess it's possible to set up some sort of natural-selection-driven environment in which AIs compete with each other; I wouldn't expect such a thing to be highly sensitive to whether we're able to capture all of the details of e.g. how to get food that occurred in our past. (I would expect it to be compute-intensive, though.)

Hopefully that gives a sense of where I'm coming from. Overall, I think this is one of the most compelling objections to Bio Anchors; I find it stronger than the points Eliezer focuses on above (unless you are pretty determined to steel-man any argument along the lines of "Brains and AIs are different" into a specific argument about the most important difference.)

Load More