Computation requires interpretation, and depends on interpretation. For example, a tap can be interpreted as a computer, that solves hydrodynamics equations. Or not. It depends on what meaning we assign to the results it produces. This flexibility of interpretation allows us to twist computations in some funny ways. Let me...
Let's say a system receives reward when it believes that it's doing some good. Kind of like RL with actor-critic. Estimated Good things -> max We can do some rewriting. I'll use notation -> inc. that means incentivised to be increased. It's like direction, towards which the gradients point. Preventing_estimated_catastrophe...
You've arrived to the same conclusion as I state. I say caring about simulated minds explodes in paradixes in my thought experiment, so we probably shouldn't. You came to the same conclusion that caring aboout digital minds shouldn't be a priority through your introduced infinitesmal measure of digital minds. We're not in disagreement here.
The hypervisor creates a bijection from real numbers to virtual machines. So, at the abstraction level of the hypervisor's interface, the number of virtual machines is continuum. Nobody says that you have to think about this system only at this layer of abstraction. But at least at this layer of abstraction there are uncountably many conscious minds. So, how are you going to apply utilitarianism in this case? The only way to make utilitarianism still work in this case, is to somehow claim that those minds don't count. And if you want to say that digital minds in general count, but in this particular case they don't count infinitely - then you have to come up with some very complex ad-hoc logic. So I conclude that utilitarianism and moral patienthood of digital minds don't mix well together. So, I discard the combination "utilitarianism+digital_morality_patients". There are many remaining moral philosophies not affected by my thought experiment. Like, utilitarianism+OrchOR is unaffected, virtue ethics is unaffected, moral egoism is unaffected.
I claim that it is possible to create a program, which can be interpreted as running uncountably infinite number of simulations. Does this interpretation carry any weight for morality? Can a simulation be viewed as a conscious mind? These questions have different answers in different philosophical frameworks. And yes, it does create weird implications in those frameworks that answer "yes" to both questions. My response is to just discard those frameworks, and use something else. What about other sizes of infinity? I don't know. I expect that it is possible to construct such a hypervisor for any size of infinity, but I'm not quite interested in doing it, because I've already discarded those philosophical frameworks in which it's important.
I claim that it has an implication that utilitarianism is not compatible with moral patienthood of digital minds. So one has to choose - either utilitarianism, or welfare of digital minds, but not both. Because otherwise, we get that every second that we didn't dedicate to building infinite number of happy minds is infinitely bad, and after we created infinite number of happy minds, utilitarianism doesn't give any instructions on how to behave, because we're already infinitely saint, and practically no action can change our total saintness score, which is absurd. There are multiple ways out of it: first, if you want to keep utilitarianism, then you can define moral patienthood in... (read more)
Computation requires interpretation, and depends on interpretation. For example, a tap can be interpreted as a computer, that solves hydrodynamics equations. Or not. It depends on what meaning we assign to the results it produces. This flexibility of interpretation allows us to twist computations in some funny ways.
Let me introduce a segment virtual machines hypervisor. It works this way. It has infinite number of virtual machines. Each VM has an address. Each VM address is a real number. The hypervisor can process commands like this: start up all machines in segment [a,b], for all machines in segment [a, b] execute command xyz. How can this hypervisor can work under the hood? It's... (read 344 more words →)
Good point. Mathematically I'd say this: there are actually a lot of competing alternative theories. "almost nothing ever happens" - is also a competing theory. From Solomonoff's induction we know that
P(event|history) = integral_{all_theories} P(event|theory)*P(history|theory)P(theory) d theory
it basically means, that we should weight each theory by the factor P(history|theory) - probability of our entire history of past observations given the theory.
What you're saying, is that if a theory is very precise, then P(history|theory) will only be high if history matches theory very well. This is why imprecise theories will have bigger weight than precise, but wrong theories. Theory "almost nothing ever happens" is very imprecise, but it is exactly why its factor P(history|theory) will often be bigger than the weight of a precise but incorrect theory. I guess normies grasp it intuitively.
Let's say a system receives reward when it believes that it's doing some good. Kind of like RL with actor-critic.
Estimated Good things -> max
We can do some rewriting. I'll use notation -> inc. that means incentivised to be increased. It's like direction, towards which the gradients point.
Preventing_estimated_catastrophe -> inc
Estimated_p_catastrophe * estimated_magnitude_of_catastroph-> inc
(P_catastrophe + p_estimation_error) * (catastrophe_magnitude + magnitude_estimation_error) -> inc
p_estimation_error -> inc
magnitude_estimation_error -> inc.
Estimation_error = k*estimation_uncertainty
estimation_uncertainty -> inc.
So, what we got: system is incentivised to have some biases:
1. It's biased to overestimate the probability and magnitude of a catastrophe.
2. It's biased to take actions in areas, where uncertainty is higher (because higher uncertainty gives more freedom to increase the first bias).
Or in... (read more)
You have oversimplified vision on rationality of humanity. You see decisions that are harmful for humanity and conclude that they are irrational. But this logic only works under the assumption that humanity is one individual. Decisions that are harmful for humanity are in most cases beneficial to the decision-making person, and therefore they are not irrational - they are selfish. This gives us much more hope, because persuading a rational selfish person with logic is totally possible.
Oh look, if we definitely the complexity as "the date when hypothesis was published", then I can say that the prior probability that our earth stands on top of a whale, on top of a turtle on top of an elephant is the highest, because this hypothesis is the oldest. And the Occam's razor becomes "don't propose new hypotheses". Trinitrotrololol)
I find it funny, that it works even in continuous case: suppose that we have probability density defined in R^n (or any other set). Then whatever bijection F:R <--> R^n we apply, the integral of probability density on that path should converge, therefore p(F(x)) goes to zero faster than 1/x. :)
Also, look: suppose the "real" universe is a random point x from some infinite set X. Let's say we are considering finite set of hypotheses "H". Probability that random hypothesis h € H is closest to x is 1/|H|. So the larger H is, the less likely it is that any particular point from it is the best description of our universe! Which gives us Occam's razor in terms of accuracy, instead of correctness, and works for uncountable sets of universes.
And in this case it is almost surely impossible to describe universe in a finite amount of symbols.
You're correct that this is what happens at one of the abstraction layers. But the choice of that layer is pretty arbitrary. By abstraction layers:
L1: hypervisor interface: uncountably many VMs
L2: hypervisor implementation: countably many VMs
L3: semiconductors: no VMs, only high and low signals
L4: electrons: no high and low signals, only electromagnetic fields
So yes, on L2 the number of VMs is finite. But why morality should count what happens on L2 and not on L1 or L3, L4? This is too arbitrary.