Emergency still feels like a "nonapple". You are right that mass is not an emergent property of quarks, but still, pretty much everything else in this universe is. If I understand it correctly, even "the distance between two specific quarks" is already an emergent property of quarks, because neither of those two quarks contains their distance in itself. So if I say e.g. "consciousness is an emergent property of quarks", I pretty much said "consciousness is not mass", which is technically true, but still mostly useless. Most of us already expected that.
Similarly, "consciousness is an emergent property of neurons" is only a surprise to those people who expected individual neurons to be conscious. I am sure such people exist. But for the rest of us, what new information does it convey?
Because the trick is that even if you don't believe that individual neurons are conscious, hearing "consciousness is an emergent property of neurons" still feels like new information. Except, there is nothing more there, only the aura of having an explanation.
The ability to express basic nonsurprising facts is useful.
When discussing whether or not to allow abortion of a fetus it matters whether you believe that real human consciousness needs a certain amount of neurons to emerge.
Plenty of people believe in some form of soul that's a unit that creates consciousness. Saying that it's emergent means that you disagree.
According to Scott's latest post about EA global, there are people at the foundational research institute who do ask themseves whether particles can be conscious.
There are plenty of cases where people try to find reductionist ways to thinking about a domain. Calories in, calories out is a common paradigm that drives a lot of thinking about diet. If you instead have a paradigm that centeres around a cybernetic system that has an emergent set point that's managed by a complex net of neurons, that paradigm gives a different perspective about what to do about weightloss.
Maybe this is just me, but it seems to me like there is a "motte and bailey" game being played with "emergence".
The "motte" is the definition provided here by the defenders of "emergence". An emergent property is any property exhibited by a system composed of pieces, where no individual piece has that property alone. Taking this literally, even "distance between two oranges" is an emergent property of those two oranges. I just somehow do not remember anyone using that word in this sense.
The "bailey" of "emergence" is that it is a mysterious process, which will somehow inevitably happen if you put a lot of pieces together and let them interact randomly. It is somehow important for those pieces to not be arranged in any simple/regular way that would allow us to fully understand their interaction, otherwise the expected effect will not happen. But as long as you close your eyes and arrange those pieces randomly, it is simply a question of having enough pieces in the system for the property to emerge.
For example, the "motte" of "consciousness is an emergent property of neurons" is saying that one neuron is not conscious, but there are some systems of neurons (i.e. brains) which are conscious.
The "bailey" of "consciousness is an emergent property of neurons" is that if you simulate a sufficiently large number of randomly connected neurons on your computer, the system is fated to evolve consciousness. If the consciousness does not appear, it must be because there are not enough neurons, or because the simulation is not fast enough.
In other words, if we consider the space of all possible systems composed of 10^11 neurons, the "motte" version merely says that at least one such system is conscious, while the "bailey" version would predict that actually most of them are conscious, because when you have sufficient complexity, the emergent behavior will appear.
The relevance for LW is that for a believer in "emergence", the problem of creating artificial intelligence (although not necessarily friendly one) is simply a question of having enough computing power to simulate a sufficiently large number of neurons.
There are positions between those. Medium-strength emergentism would have it that some systems are conscious, that cosnciousness is not a property of their parts, and that it is not reductively understandable in terms of the parts and their interactions, but that it is by no mean inevitable.
Reduction has its problems too. Many writings on LW confuses the claim that things are understandable in terms of their parts with the claim that they are merely made of parts.
Eg:-
(1) The explanatory power of a model is a function of its ingredients. (2) Reductionism includes all the ingredients that actually exist in the real world. Therefore (3) Emergentists must be treating the “emergent properties” as extra ingredients, thereby confusing the “map” with the “territory”. So Reductionism is defined by EY and others as not treating emergent properties as extra ingredients (in effect).
I am questioning the implicit premise that some kinds of emergent things are "reductively understandable in terms of the parts and their interactions." I think humans have a basic problem with getting any grasp at all on the idea of things being made of other things, and therefore you have arguments like those of Parmenides, Zeno, etc., which are basically a mirror of modern arguments about reductionism. I would illustrate this with Viliam's example of the distance between two oranges. I do not see how the oranges explain the fact that they have a distance between them, at all. Consciousness may seem even less intelligible, but this is a difference of degree, not kind.
I am questioning the implicit premise that some kinds of emergent things are "reductively understandable in terms of the parts and their interactions.
It's not so much some emergent things, for a uniform definiton of "emergent", as all things that come under a vriant definition of "emergent".
I think humans have a basic problem with getting any grasp at all on the idea of things being made of other things, and therefore you have arguments like those of Parmenides, Zeno, etc., which are basically a mirror of modern arguments about reductionism
Not really, they are about what we would now call mereology. But as I noted, the two tend to get conflated here.
. I would illustrate this with Viliam's example of the distance between two oranges. I do not see how the oranges explain the fact that they have a distance between them, at all.
Reductionism is about preserving and operating within a physicalist world view, and physicalism is comfortable with spacial relations and causal interactions as being basic elements or reality. Careful reducitonists say "reducible to its parts, their structure, and their interactions".
"physicalism is comfortable with spacial relations and causal interactions as being basic elements or reality"
I am suggesting this is a psychological comfort, and there is actually no more reason to be comfortable with those things, than with consciousness or any other properties that combinations have that parts do not have.
I don't understand what is supposed to be so bad about "mysterious" things. Take the distance between two oranges: if you look at a single orange, it doesn't tell you anything about how far it should be from another. And special relativity implies that there is no difference between a situation where one orange is moving and the other isn't, and the situation where the movements are reversed. So the distance between two oranges can be changing, even though apparently neither one is changing more than the other, or at all, when you just sit and look at one of the oranges. So the distance between two oranges seems pretty mysterious to me.
Also, I'm not sure anyone actually says that emergent things "inevitably happen" due to a large quantity and randomness.
Like many cases of Motte-and-Bailey, the Motte is mainly held by people who dislike the Bailey. I suspect that an average scientist in a relevant field somewhere at or below neurophysics in the generality hierarchy (e.g. chemist, physicist, but not sociologist), would consider that bailey to be… non-likely at best, while holding the motte very firmly.
The relevance for LW is that for a believer in "emergence", the problem of creating artificial intelligence (although not necessarily friendly one) is simply a question of having enough computing power to simulate a sufficiently large number of neurons.
I don't think in practice that has much to do with whether or not someone uses the word emergence. As far as a I understand EY thinks that if you simulate enough neurons sufficiently well you get something that's conscious.
I understand EY thinks that if you simulate enough neurons sufficiently well you get something that's conscious.
Without specifying the arrangements of those neurons? Of course it should if you copy the arrangement of neurons out of a real person, say, but that doesn't sound like what you meant.
I think you're right. I also think saying 'x is emergent' may sound more magical than it is, if I am understanding emergence right, depending on your understanding of it. Like it doesn't mean that the higher scale phenomenon isn't /made up of/ lower-level phenomena, but that it isn't (like a homonculi) itself present as anything smaller than that level. Like a robot hopping kangaroo toy needs both a body, and legs. The hopping behavior isn't contained in the body - that just rotates a joint. The hopping behavior isn't contained in the legs - those just have a joint that can connect to the body joint. Its only when the two bits are plugged into each other that the 'hopping' behavior 'emerges' from the torso-legs system. Its not coming from any essential 'hoppiness' in the legs or the torso. I think it can seem a bit magical because it can sound like the behavior just 'appears' at a certain point but its no more than a picture of a tiger 'appears' from a bunch of pixels. Only we're talking about names for systems of functions (hopping is made of the leg and torso behaviors and their interaction with the ground and stuff) more than names for systems of objects (tiger picture is made up of lines and corners and stuff are made of pixels and stuff). In some sense 'tigers' and 'hopping' don't really exist - just pixels (or atoms or whatever) and particle interactions. But we have names for systems of objects, and systems of functions, because those names are useful.
Yeah, your take seems right, and agrees with Wikipedia and SEP.
Though I'm not sure it's worth your time to correct Eliezer's mistakes so painstakingly. He made lots of them. The biggest ones were probably betting against academia and betting against neural networks. His attractiveness as a writer comes in part from overconfidence, but in the real world a couple big mistakes from overconfidence can wipe out all most of your wins from it.
I disagree. Performing this sort of work is part of engaging with the ideas. It's maybe not that interesting to you now, sure, but I've written similar things in the past in the process of building my understanding of ideas.
Which bets did he make against academia?
When I eventually become an AI researcher, I do plan to try another approach apart from Neural networks though (I have an idea I think might work, and enough people are already trying their hands at neural nets).
I agree that I do find Eliezer's overconfidence endearing.
I don't know. I have a plan for a start up, if it succeeds, I probably wouldn't go. If the plan fails, I'll probably go to grad school.
I plan to start full AI research at earliest 25, and at latest 30. (I'm 19 now).
I think he solved the college problem by other means (recruiting people who are credible to bridge to the academic community). Not sure this was a mistake, considering costs like "not writing HPMOR". Yes I agree some of the attitudes have produced unnecessary social friction, (and maybe a lot of it) but its harder to claim the individual decision was obviously bad.
MIRI publishes very few peer reviewed papers per researcher per year, so the mistake of betting against academia is still hurting it.
Granted. Do you think the lack of peer review is hurting via the papers not getting enough acceptance in the community that would add to the AI safety cause? Or is the quality of papers themselves lacking due to not aiming at the peer review standard?
I think everything would've been better (more work, more discussion, more credibility) if the whole thing had been in academia from the beginning.
Another unfortunate factor was Eliezer's crazy emphasis on secrecy in the early days.
Academia can be terribly inefficient though. There were hundreds of academic papers on electronic cash before Bitcoin and none of them went anywhere in terms of deployment and commercial success, nor did any of them propose something architecturally close to Bitcoin. Similarly there were dozens of academic papers on the Absent Minded Driver problem but all that work didn't get very far. Since lots of very smart people go into academia, it seems that there's something wrong with the incentives in academia that prevents people from making the kind of progress that we've seen on these topics outside of academia. If Eliezer went into academia he may have fallen prey to the same kinds of problems.
I think your argument would work better when flipped on its head. There's something wrong with incentives outside academia that prevents people from coming up with the Absent Minded Driver problem, and many other things besides! Most of the prerequisites for both Bitcoin and UDT come from academia.
Very few people outside academia has resources to do research so if you look at the total sum of results academia is obviously going to look better. But if you compare results per unit of work or cost, it no longer looks so great. Consider that the few people who worked on digital cash outside of academia quickly converged to ideas resembling Bitcoin, while none of the many academics did even after spending many more years on the problem. Similarly with UDT. It seems to me that if Eliezer went into academia, there's no reason to think he would be much more productive than the average academic decision theorist. Nor would he likely have created a forum for amateurs to discussion decision theory (since no actual academic decision theorist has), in which case I wouldn't have found an outlet for to spread my own decision theory ideas.
And to clarify, in case this is why you're disagreeing with me, obviously academia is very good at certain types of problems, for example those involving incremental technical progress (like making slightly stronger ciphers each year until they're really efficient and practically unbreakable). So I'm only talking about a subset of fields that apparently includes digital money and decision theory.
It doesn't look like a subset of fields to me. More like a subset of steps in each field that need to be done by outsiders, while both preceding and following steps can be done by academia. That's what I'm seeing with decision theory: academia worked out many prerequisites but got stuck, then LW got the chance to make a key step, but it didn't feed back into academia so we've stagnated since then.
In both digital money and decision theory, academia was going in the wrong direction, in the sense that you did worse if you tried to build on top of the state of the art in academia instead of starting over from a past point. In digital money you essentially had to go back to just basic cryptographic primitives. (I can't think of anything in the hundreds of academic papers on digital money that Bitcoin uses.) In decision theory the state of the art was CDT, and one of the justifications for CDT was that, to be rational, one ought to two-box in Newcomb's problem and CDT does that while EDT doesn't. I recall trying to come up with a version of UDT that would two-box in Newcomb's problem and play D in PD against a copy because (until reading Eliezer's posts on LW) I just took for granted without thinking about it that CDT must be an improvement over EDT and the academic consensus on Newcomb's problem must be correct. I think I posted the resulting decision theory (which ended up being a lot more complex than the UDT I posted LW) to the "everything-list" and I can try to find it if you'd like.
We also don't have evidence that following steps can be done by academia. In both Bitcoin and UDT the following steps are all being done outside of academia, as far as I can tell. I don't think "we've stagnated" is right since the obvious next step after UDT is trying to solve logical uncertainty so we can tell what UDT actually recommends, and progress is being made there.
Even if we put this dispute aside, and just assume that key steps in each field need to be done by outsiders, that seems enough to conclude that we wouldn't be better off now if Eliezer had gone into academia. If that had happened, there wouldn't be anyone to catalyze the crucial "outsider step" and we'd still be stuck with CDT (and the overly complex and wrong version of UDT) today.
On second thought, my previous reply was a bit too hasty and combative. What you said does make sense; it's perhaps more likely that academia is unable to take certain steps rather than that it can't make any progress in entire fields. What I wrote about academia going in wrong directions is consistent with the missing outsider steps having been missing for a while, and academia not recognizing it.
If I steelman your argument from "Eliezer should have gone into academia" to "Eliezer should have made a greater effort to push his results back into academia" it becomes harder to refute, but still not obviously right. It seems that if academia is unable to take certain steps, it's also unable to recognize that it's missing those steps and also unable to recognize when outsiders have taken those steps. It's not clear what resources (in terms of, e.g., skills and connections) an outsider would need to wield in order to force academia past those missing steps and to continue to make progress, and also unclear how far academia can go before it encounters another step that only outsiders can take.
Both Satoshi and Eliezer did make nontrivial efforts to let academia recognize their results. Satoshi by posting an academic-looking paper and Eliezer by working with someone to try to polish his results into acceptable academic form. Given that they failed, I'm tempted to think that it's just not realistic to expect that someone exists with the necessary combination of research skills need to make crucial progress, personality or circumstances causing them to stay outside of academia, and resources needed to force academia to recognize their progress.
Do you know if Satoshi's bitcoin whitepaper or Eliezer's TDT writeup were ever submitted for peer review? If not, then it seems premature to say that academia has rejected them.
I don't have that information, but being published in a peer-reviewed venue is not a prerequisite for academia to recognize a result. See this example. Getting a paper through peer review is very costly in terms of time and effort (especially at a highly reputable journal which might only accept less than 10% of submissions and that's from academics who know what they're supposed to do to maximize the chances of being accepted), and may not buy all that much in terms of additional attention from the people who might be able to build upon the work. I tried submitting a paper to an academic crypto conference (for a crypto primitive, not b-money) so I have some experience with this myself. Satoshi and Eliezer aren't the only very smart people who haven't tried very hard to publish in academia (i.e., hard enough to get published in a reputable journal). Just from people I know, there's also Gary Drescher, Paul Christiano (for his approval-directed agent ideas), and other researchers at MIRI who seem to publish most of their results as technical reports. I guess that also includes you, who haven't followed your own advice?
Obviously whole fields of academia going in wrong directions represents a huge societal waste, and it would be great to have a solution to fix that; I'm just not sure what the solution is. (Note that among other things, you have to get academics to admit at least implicitly that for decades they've been wasting time and other people's money on wrong approaches.) I haven't been too concerned about this for decision theory since I'm not sure that further progress in decision theory is really crucial for (or even contributes positively to) AI alignment, but I have been thinking about how to get more academics to switch their focus from ML and AI capability in general to AI alignment, especially to AI alignment ideas that we think are promising and neglected (like Paul's ideas). So far my idea is to try to influence funders (Open Phil, FLI, other philanthropists) to direct their grants to those specific ideas.
I guess that also includes you, who haven't followed your own advice?
Yeah. When I was working actively on MIRI math, I picked up the idea that getting stuff peer reviewed is nice but not necessary for progress. My opinion changed in the last couple years, when I was already mostly away. The strategy I'd suggest now is to try to join the academic conversation on their terms, as I did at the Cambridge conference. Ideally, getting publications in journals should be part of that.
I haven't thought much about talking to funders, good to hear you're pursuing that.
My opinion changed in the last couple years
What triggered this?
The strategy I'd suggest now is to try to join the academic conversation on their terms, as I did at the Cambridge conference.
MIRI seems to be doing more of this as well, but I'm not seeing any noticeable results so far. Judging by citations in Google Scholar, in the 2 years since that conference, it doesn't look like any academics have picked up on the ideas presented there by you and MIRI people or made further progress?
One other thing that worries me is, unless we can precisely diagnose what is causing academia to be unable to take the "outsider steps", it seems dangerous to make ourselves more like academia. What if that causes us to lose that ability ourselves?
I haven't thought much about talking to funders, good to hear you're pursuing that.
Well I'm doing what I can but I'm not sure I'm the best person for this job, given that I'm not very social/outgoing and my opportunities for travel are limited so it's hard to meet those funders and build up relationships.
One other thing that worries me is, unless we can precisely diagnose what is causing academia to be unable to take the "outsider steps", it seems dangerous to make ourselves more like academia. What if that causes us to lose that ability ourselves?
Seems that academic motivations can be "value", e.g discovering something of utility or "momentum", sort of like a beauty contest, more applicable in abstract areas where utility is not obvious. Possible third is immediate enjoyment which probably contributed to millennia of number theory before it became useful.
Doing novel non-incremental things for non-value (like valuing AI safety) reasons is likely to be difficult until enough acceptability is built up for momentum type motivations. (which also suggests trying to explicitly build up momentum as an intervention)
MIRI did try to hire an academic to get TDT peer reviewed. It didn't work out but I don't think "refuse" is a good description.
You don't need to hire an academic to get peer review. Just send it to a relevant journal, and because of the blind review system, they will review it. I did it myself and got many valuable comments on my articles. One was finally accepted and one was rejected.
Writing academic papers needs a certain skillset. That's not a skillset that EY developed, so it makes sense to give that task to someone who has it.
I think it is trainable skillset, and EY also has some papers, so I think he is able to master it. But he decided to write texts which are more attention grabbing.
As far as I understand the papers with EY in the authors list are collaborations between EY and other people where a lot of ideas come from EY but the actual paper writing was done by someone else.
Well, size and mass of particles? I would NOT DARE diving into this... certainly not in front of any string theorist (OK, ANY physics theorist, and not only). Even space can easily turn out to be "emergent" ;-).