Here's a blog post arguing that conceptual analysis has been a complete failure, with a link to a paper saying the same thing: http://fakenous.net/?p=1130
Sure, but within AI, intelligence is the main feature that we're trying very hard to increase in our systems that would plausibly let the systems we build outcompete us. We aren't trying to make AI systems that replicate as fast as possible. So it seems like the main thing to be worried about is intelligence.
Blaise Agüera y Arcas gave a keynote at this NeurIPS pushing ALife (motivated by specification problems, weirdly enough...: https://neurips.cc/Conferences/2019/Schedule?showEvent=15487).
The talk recording: https://slideslive.com/38921748/social-intelligence. I recommend it.
With 0, the AI never does anything and so is basically a rock
I'm trying to point at "myopic RL", which does, in fact, do things.
You might object that all of these can be made state-dependent, but you can make your example state-dependent by including the current time in the state.
I do object, and still object, since I don't think we can realistically include the current time in the state. What we can include is: an impression of what the current time is, based on past and current observations. There's an epistemic/indexical problem here you're ignoring.
I'm not an expert on AIXI, but my impression from talking to AIXI researchers and looking at their papers is: finite-horizon variants of AIXI have this "problem" of time-inconsistent preferences, despite conditioning on the entire history (which basically provides an encoding of time). So I think the problem I'm referring to exists regardless.
Can't I say that the emulation is me, and does morally matter (via FCToM), and also the many people enslaved in the system morally matter (via regular morality)?
What I'm saying is that, to use the language of the debate I referenced, "what kind of paper the equation is written on DOES matter".
It seems like you're saying that FCToM implies that if a physical system implements a morally relevant mathematical function, then the physical system itself cannot include morally relevant bits, and I don't see why that has anything to do with FCToM.
I'm saying "naive FCToM", as I've characterized it, says that. I doubt "naive FCToM" is even coherent. That's sort of part of my broader point (which I didn't make yet in this post).
I think I was maybe trying to convey too much of my high-level views here. What's maybe more relevant and persuasive here is this line of thought:
Also, nitpicking a bit: to a large extent, society is trying to make systems that are as competitive as possible at narrow, profitable tasks. There are incentives for excellence in many domains. FWIW, I'm somewhat concerned about replicators in practice, e.g. because I think open-ended AI systems operating in the real-world might create replicators accidentally/indifferently, and we might not notice fast enough.
My main opposition to this is that it's not actionable
I think the main take-away from these concerns is to realize that there are extra risk factors that are hard to anticipate and for which we might not have good detection mechanisms. This should increase pessimism/paranoia, especially (IMO) regarding "benign" systems.
Idk, if it's superintelligent, that system sounds both rational and competently goal-directed to me.
(non-hypothetical Q): What about if it has a horizon of 10^-8s? Or 0?
I'm leaning on "we're confused about what rationality means" here, and specifically, I believe time-inconsistent preferences are something that many would say seem irrational (prima face). But
Perhaps you mean grey-goo type scenarios where we wouldn't call the replicator "intelligent", but it's nonetheless a good replicator? Are you worried about AI systems of that form? Why?
Yes, I'm worried about systems of that form (in some sense). The reason is: I think intelligence is just one salient feature of what makes a life-form or individual able to out-compete others. I think intelligence, and fitness even more so, are multifaceted characteristics. And there are probably many possible AIs with different profiles of cognitive and physical capabilities that would pose an Xrisk for humans.
For instance, any appreciable quantity of a *hypothetical* grey goo that could use any matter on earth to replicate (i.e. duplicate itself) once per minute would almost certainly consume the earth in less than one day (I guess modulo some important problems around transportation and/or its initial distribution over the earth, but you probably get the point).
More realistically, it seems likely that we will have AI systems that have some significant flaws but are highly competent at strategically relevant cognitive skills, able to think much faster than humans, and have very different (probably larger but a bit more limited) arrays of sensors and actuators than humans, which may pose some Xrisk.
The point is just that intelligence and rationality are import traits for Xrisk, but we should certainly not make the mistake of believing one/either/both are the only traits that matter. And we should also recognize that they are both abstractions and simplifications that we believe are often useful but rarely, if ever, sufficient for thorough and effective reasoning about AI-Xrisk.
Sure, I more meant competently goal-directed.
This is still, I think, not the important distinction. By "significantly restricted", I don't necessarily mean that it is limiting performance below a level of "competence". It could be highly competent, super-human, etc., but still be significantly restricted.
Maybe a good example (although maybe departing from the "restricted hypothesis space" type of example) would be an AI system that has a finite horizon of 1,000,000 years, but no other restrictions. There may be a sense in which this system is irrational (e.g. having time-inconsistent preferences), but it may still be extremely competently goal-directed.
At a meta-level: this post might be a bit to under-developed to be worth trying to summarize in the newsletter; I'm not sure.
RE the summary:
RE the opinion:
OK I understand. JTBC, my original statement was: " language and human culture created lead to a massive increase in returns to intelligence", not that larger brains (/greater intelligence) suddenly became valuable.
TBC: I'm alluding to others' scholarly arguments, which I'm not very familiar with. I'm not sure to what extent these arguments have empirical vs. theoretical basis.
Basic question: how can this be a sufficient explanation? There needs to be *some* advantage to having the bigger brain, it being "cheap" isn't a good enough explanation...