All of jaan's Comments + Replies

the werewolf vs villager strategy heuristic is brilliant. thank you!

Credit to Benquo's writing for giving me the idea.

if i understand it correctly (i may not!), scott aaronson argues that hidden variable theories (such as bohmian / pilot wave) imply hypercomputation (which should count as an evidence against them):

If hypercomputation is defined as computing the uncomputable, then that's not his idea. It's just a quantum speedup better than the usual quantum speedup (defining a quantum complexity class DQP that is a little bigger than BQP). Also, Scott's Bohmian speedup requires access to what the hidden variables were doing at arbitrary times. But in Bohmian mechanics, measuring an observable perturbs complementary observables (i.e. observables that are in some kind of "uncertainty relation" to the first) in exactly the same way as in ordinary quantum mechanics.  There is a way (in both Bohmian mechanics and standard quantum mechanics) to get at this kind of trajectory information, without overly perturbing the system evolution - "weak measurements". But weak measurements only provide weak information about the measured observable - that's the price of not violating the uncertainty principle. A weak measuring device is correlated with the physical property it is measuring, but only weakly.  I mention this because someone ought to see how it affects Scott's Bohmian speedup, if you get the history information using weak measurements. (Also because weak measurements may have an obscure yet fundamental relationship to Bohmian mechanics.) Is the resulting complexity class DQP, BQP, P, something else? I do not know. 

interesting, i have in my reading queue for a few years now -- i take your comment as an upvote!

myself i swear by FDT (somewhat abstract, sure, but seems to work well) and freestyle dancing (the opposite of abstract, but also seems to work well). also coding (eg, just spent several days using pandas to combine and clean up my philanthropy data) -- code grounds one in reality.

having seen the “kitchen side” of the letter effort, i endorse almost all zvi’s points here. one thing i’d add is that one of my hopes urging the letter along was to create common knowledge that a lot of people (we’re going to get to 100k signatures it looks like) are afraid of the thing that comes after GPT4. like i am.

thanks, everyone, who signed.

EDIT: basically this:

while it’s easy to agree with some abstract version of “upgrade” (as in try to channel AI capability gains into our ability to align them), the main bottleneck to physical upgrading is the speed difference between silicon and wet carbon:

5Jed McCaleb6mo
Yeah to be clear I don't think "upgrading" is easy. It might not even be possible in a way that makes it relevant. But I do think it offers some hope in an otherwise pretty bleak landscape.

yup, i tried invoking church-turing once, too. worked about as well as you’d expect :)

looks great, thanks for doing this!

one question i get every once in a while and wish i had a canonical answer to is (probably can be worded more pithily):

"humans have always thought their minds are equivalent to whatever's their latest technological achievement -- eg, see the steam engines. computers are just the latest fad that we currently compare our minds to, so it's silly to think they somehow pose a threat. move on, nothing to see here."

note that the canonical answer has to work for people whose ontology does not include the concepts of "computation"... (read more)

Most of the threat comes from the space of possible super-capable minds that are not human. (This does not mean that human-like AIs would be less dangerous, only that they are a small part of the space of possibilities.)
2Ben Livengood8mo
Agents are the real problem. Intelligent goal-directed adversarial behavior is something almost everyone understands whether it is other humans or ants or crop-destroying pests. We're close to being able to create new, faster, more intelligent agents out of computers.
2Lone Pine8mo
I think the technical answer comes down to the Church-Turing thesis and the computability of the physical universe, but obviously that's not a great answer for the compscidegreeless among us.

the potentially enormous speed difference ( will almost certainly be an effective communications barrier between humans and AI. there’s a wonderful scene of AIs vs humans negotiation in william hertling’s “A.I. apocalypse” that highlights this.

i agree that there's the 3rd alternative future that the post does not consider (unless i missed it!):

3. markets remain in an inadequate equilibrium until the end of times, because those participants (like myself!) who consider short timelines remain in too small minority to "call the bluff".

see the big short for a dramatic depiction of such situation.

great post otherwise. upvoted.


Coincidentally, that scene in The Big Short takes place on January 11 (2007) :D

yeah, this seems to be the crux: what will CEV prescribe for spending the altruistic (reciprocal cooperation) budget on. my intuition continues to insist that purchasing the original star systems from UFAIs is pretty high on the shopping list, but i can see arguments (including a few you gave above) against that.

oh, btw, one sad failure mode would be getting clipped by a proto-UFAI that’s too stupid to realise it’s in a multi-agent environment or something,

ETA: and, tbc, just like interstice points out below, my “us/me” label casts a wider net than “us in this particular everett branch where things look particularly bleak”.

roger. i think (and my model of you agrees) that this discussion bottoms out in speculating what CEV (or equivalent) would prescribe.

my own intuition (as somewhat supported by the moral progress/moral circle expansion in our culture) is that it will have a nonzero component of “try to help out the fellow humans/biologicals/evolved minds/conscious minds/agents with diminishing utility function if not too expensive, and especially if they would do the same in your position”.

tbc, i also suspect & hope that our moral circle will expand to include all fellow sentients. (but it doesn't follow from that that paying paperclippers to unkill their creators is a good use of limited resources. for instance, those are resources that could perhaps be more efficiently spent purchasing and instantiating the stored mindstates of killed aliens that the surviving-branch humans meet at the edge of their own expansion.) but also, yeah, i agree it's all guesswork. we have friends out there in the multiverse who will be willing to give us some nice things, and it's hard to guess how much. that said, i stand by the point that that's not us trading with the AI; that's us destroying all of the value in our universe-shard and getting ourselves killed in the process, and then banking on the competence and compassion of aliens. (in other words: i'm not saying that we won't get any nice things. i'm saying that the human-reachable fragment of the universe will be ~totally destroyed if we screw up, with ~none of it going to nice things, not even if the UFAI uses LDT.)

yeah, as far as i can currently tell (and influence), we’re totally going to use a sizeable fraction of FAI-worlds to help out the less fortunate ones. or perhaps implement a more general strategy, like mutual insurance pact of evolved minds (MIPEM).

this, indeed, assumes that human CEV has diminishing returns to resources, but (unlike nate in the sibling comment!) i’d be shocked if that wasn’t true.

one thing that makes this tricky is that, even if you think there's a 20% chance we make it, that's not the same as thinking that 20% of Everett branches starting in this position make it. my guess is that whether we win or lose from the current board position is grossly overdetermined, and what we're fighting for (and uncertain about) is which way it's overdetermined. (like how we probably have more than one in a billion odds that the light speed limit can be broken, but that doesn't mean that we think that one in every billion photons breaks the limit.) the surviving humans probably don't have much resource to spend, and can't purchase all that many nice things for the losers. (Everett branches fall off in amplitude really fast. Exponentially fast. Back-of-the-envelope: if we're 75 even-odds quantum coincidences away from victory, and if paperclipper utility is linear in matter, then the survivors would struggle to purchase even a single star for the losers, even if they paid all their matter.) ftr, i'm pretty uncertain about whether CEV has diminishing returns to resources on merely cosmic scales. i have some sympathy for arguments like vanessa's, and it seems pretty likely that returns diminish eventually. but also we know that two people together can have more than twice as much fun as two people alone, and it seems to me that that plausibly also holds for galaxies as well. as a stupid toy model, suppose that every time that population increases by a factor of ten, civilization's art output improves by one qualitative step. and suppose that no matter how large civilization gets, it factors into sub-communities of 150 people, who don't interact except by trading artwork. then having 10 separate universes each with one dunbar cluster is worse than having 1 universe with 10 dunbar clusters, b/c the latter is much like the former except that everybody gets to consume qualitatively better art. separately, it's unclear to me whether humanity, in the fragment of w

sure, this is always a consideration. i'd even claim that the "wait.. what about the negative side effects?" question is a potential expected value spoiler for pretty much all longtermist interventions (because they often aim for effects that are multiple causal steps down the road), and as such not really specific to software.

great idea! since my metamed days i’ve been wishing there was a prediction market for personal medical outcomes — it feels like manifold mechanism might be a good fit for this (eg, at the extreme end, consider the “will this be my last market if i undertake the surgery X at Y?” question). should you decide to develop such aspect at one point, i’d be very interested in supporting/subsidising.

3Austin Chen2y
Yes, that's absolutely the kind of prediction market we'd love to enable at Manifold! I'd love to chat more about specifically the personal medical use case, and we'd already been considering applying to SFF -- let's get in touch (I'm

actually, the premise of david brin’s existence is a close match to moravec’s paragraph (not a coincidence, i bet, given that david hung around similar circles).

confirmed. as far as i can tell (i’ve talked to him for about 2h in total) yi really seems to care, and i’m really impressed by his ability to influence such official documents.

indeed, i even gave a talk almost a decade ago about the evolution:humans :: humans:AGI symmetry (see below)!

what confuses me though is that "is general reasoner" and "can support cultural evolution" properties seemed to emerge pretty much simultaneously in humans -- a coincidence that requires its own explanation (or dissolution). furthermore, eliezer seems to think that the former property is much more important / discontinuity causing than the latter. and, indeed, outsized progress being made by individual human reasoners (scientists/inventors/etc.) see... (read more)

1Gram Stone2y
If information is 'transmitted' by modified environments and conspecifics biasing individual search, marginal fitness returns on individual learning ability increase, while from the outside it looks just like 'cultural 'evolution.''
4Vanessa Kosoy2y
I think that these properties encourage each other's evolution. When you're a more general reasoner, you have a bigger hypothesis space, specifying a hypothesis requires more information, so you also benefit more from transmitting information. Conversely, once you can transmit information, general reasoning becomes much more useful since you effectively have access to much bigger datasets.
David Deutsch (in The Beginning of Infinity) argues, as I recall, that they're basically the same faculty. In order to copy someone else / "carry on a tradition", you need to model what they're doing (so that you can copy it), and similarly for originators to tell whether students are correctly carrying on the tradition. The main thing that's interesting about his explanation is how he explains the development of general reasoning capacity, which we now think of as a tradition-breaking faculty, in the midst of tradition-promoting selection. If you buy that story, it ends up being another example of treacherous turn from human history (where individual thinkers, operating faster than cultural evolution, started pursuing their own values).

amazing post! scaling up the community of independent alignment researchers sounds like one of the most robust ways to convert money into relevant insights.

indeed they are now. retrocausality in action? :)


well, i've always considered human life extension as less important than "civilisation's life extension" (ie, xrisk reduction). still, they're both very important causes, and i'm happy to support both, especially given that they don't compete much for talent. as for the LRI specifically, i believe they simply haven't applied to more recent SFF grant rounds.