No Evolutions for Corporations or Nanodevices

Eliezer, the criteria you list may be necessary for the evolution of *complex* structures. But I think it's worth highlighting that practically important evolutionary results could come about without the need for new complex structures. For example, suppose we have a population of controlled self-replicating nanobots, built unwisely in such a way that they keep replicating until a separate breaking circuit kicks in and shuts off replication. Now suppose there is a mutation in the code of one nanobot such that its offspring lack a working breaking circuit.... (read more)

Pascal's Mugging: Tiny Probabilities of Vast Utilities

Wei, no I don't think I considered the possibility of discounting people by their algorithmic complexity.

I can see that in the context of Everett it seems plausible to weigh each observer with a measure proportional to the amplitude squared of the branch of the wave function on which he is living. Moreover, it seems right to use this measure both to calculate the anthropic *probability* of me finding myself as that observer and the moral *importance* of that observer's well-being.

Assigning anthropic probabilities over infinite domains is problematic. I don't... (read more)

Pascal's Mugging: Tiny Probabilities of Vast Utilities

I have a paper which explores the problem in a somewhat more general way (but see especially section 6.3).

Infinite Ethics: http://www.nickbostrom.com/ethics/infinite.pdf

9/26 is Petrov Day

Carl, I like your suggestion to establish a prize for avoing mega-disasters and existential risks. (Meanwhile, I'm going to send Petrov a small donation.)

One of the bias issues this raises is the possibility of bias in how we allocate our attention. One could think of an attention allocation as if it involved an implicit belief that "this is worth attending to". Then we can think of how this kind of implicit belief might be biased. For example, in the ancestral environment nobody was worth attending to because they had prevented millions of de... (read more)

Mysterious Answers to Mysterious Questions

And to continue the thread of Roy's comment as picked up by Eliezer, it might have been a fairly reasonable conjecture at the time (or at some earlier time). We have to be wary about hindsight bias. Imagine a time before biochemistry and before evolution theory. The only physicalist "explanations" you've ever heard of or thought of for why animals exist and how they function are obvious non-starters...

You think to yourself, "the folks who are tempted by such explanations just don't realize how far away they are from really explaining this... (read more)

910yWe don't need to imagine. We are in exactly this position with respect to
consciousness.

One Life Against the World

I have a paper on this problem of infinities in ethics: http://www.nickbostrom.com/ethics/infinite.pdf

It is a difficult topic.

Marginally Zero-Sum Efforts

I once suggested to the EU (whose research funding application process is exceptionally laborious) that when evaluating the effectiveness of their grants scheme, they should take into account not only the internal costs of administering the scheme, but also the costs of applying. More specifically, they should estimate the salary cost of the time the applicants put into learning about the program and applying (unsuccessful applicants included) and view this as an administrative cost of the program. I think it would be an eye-opener. Of course, the chances that they would actually start doing this as a result of my suggestion are indistinguishably close to zero.

Archimedes's Chronophone

I'm not sure I understand exactly how the chronophone works. It sounds a bit like the only useful ideas a person can transmit are ideas that she herself has independently worked out or discovered; in which case not the same ideas but some analogous and similarly useful ideas gets delivered to Archimedes. In this case, I guess I might try to read out some of my research papers, hoping that they contain some useful original insights. It might also work if I transmit ideas that have originated with others but whose merits I have grasped through my own indepen... (read more)

Outside the Laboratory

Eliezer wrote: "Godel's Completeness theorem shows that any first-order statement true in all models of a set of first-order axioms is provable from those axioms. Thus, the failure of Peano Arithmetic to prove itself consistent is because there are many "supernatural" models of PA in which PA itself is not consistent; that is, there exist supernatural numbers corresponding to proofs of P&~P."

This is getting far from the topic but... I really don't see how Completeness entails anything about PA's failure to prove itself consistent (m... (read more)

"In IEEE Spectrum's sad little attempt at Singularity coverage, one bright spot is Paul Wallich's "Who's Who In The Singularity",..."

Brightness here being a relative quality... I am labeled green, meaning "true believer, thinks it will happen within 30 years." Yet I am quoted (correctly) as saying "I would... assign less than a 50% probability to superintelligence being developed by 2033." (I also don't endorse "once the singularity comes near, we will all be kicking ourselves for not having brought it about s... (read more)