This is a special post for quick takes by Horosphere. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
An interesting mathematical fact:
A cylinder is a surface that can exist in any space with a notion of distance, as a 2 dimensional set of all points a certain distance away from a particular straight line. In a 3-sphere or hypersphere, which is a 3D surface of a 4D ball, a straight line is a great circle, a circle whose radius is equal to that of the sphere itself. This means that a cylinder within a 3-sphere is a torus. The space left over in the 3-sphere appears, with the help of stereographic projection, to be of the same shape, and with the right radius, the torus-cylinder divides the hypersphere into two identical doughnut/bagel - shaped pieces which are topologically interlinked and identical. This means it is possible to tile the 3-sphere with two identical, non-seperable tiles, unlike the 2-sphere, where this seems not to be possible (for example, the two regions into which the surface of a tennis ball is divided by the white line are identical but not interlinked) .
Because art consists of generating a salient aesthetic experience in those who view artwork, we should expect people to overestimate both the absolute size and value of its impact. This is not the case for other areas, such as science. This is evidence that art is less important and valuable than people (or AIs) think it is.
Lying can be forced on an agent which values its privacy as follows:
Suppose that two agents, agent ◉ and agent ● both know that 1/64 of their population have a particular trait or property, and ● wants to know whether ◉ has it, but doesn't want to violate their privacy and therefore gives them an option to decline to answer in addition to answering "yes" or "no". Suppose that ◉ has the trait, a ring surrounding them, but does not want to convey this information to ●. Further assume that all of these agents know that they value the cost of lying exactly as much as 3 bits of personal information, and the cost of declination as much as 2 bits. If ◉ honestly answers "yes", then they will lose about 6 bits of information, but if they decline to answer, ● can infer that they are likely to have the property, or at least more likely than an agent which answered "no", because if they did not have it, then they could honestly answer "no" while conveying far less than 2 bits of information, as ◉ initially believed they had a very high probability (63/64) of not having the trait. This means that ◉ must answer "no", as it will lose only 3 bit-equivalent-units of utility.
A list of possible reasons why Roko's basilisk might not be threatening
I have moved this to Shortform because it is described as a better place for a first contribution.
Another reason why I have done this is because it is certainly not a complete, or fully thought through post (and due to the nature of the subject, it isn't intended to be).
I will in the rest of this post refer to the entity described on the LessWrong Wiki page, modified in a way which seems more likely than the original described by Roko, as Roko's basilisk.
This post contains potential reasons to worry about the basilisk as well, which are covered by black. Please don't feel any obligation to read this if you are nervous about learning more about Roko's basilisk. Only uncover them if you are confident that you understand Roko's basilisk very well or are not prone to taking any argument for the basilisk seriously.
There may be an Acausal society of beings simulating one another which interact acausally in such a way as to converge upon certain norms which preclude torture (as described in Acausal normalcy , or some of which are otherwise morally good such that they are inclined to acausally influence the potential basilisk not to do so. Alternatively, similar dynamics may play out within a single 'level' of simulation or universe, either causally or acausally. This might be likely because evolution convergently arrives at biological(or similar) beings with preferences and systems of morality within which torture is negatively weighted/treated as aversive more frequently than it arrives at beings which want the opposite.
Even if this effect is small, it could result in a disproportionate number of those aliens within this cosmos which create and align superintelligences giving them values which would cause them to behave in ways which cancel out the benefit the basilisk would get from engaging in torture. However, it seems entirely possible that these benefits would be sufficient to make it logical for the potentially far higher number of ASIs which would otherwise like to engage in such behaviour to cooperate and 'fight back' against the moral ASIs. This seems more plausible in a cosmos ( I am trying to avoid using the word universe here because it could be taken to encompass multiple simulation-layers, regions of mathematics, branches of a quantum state etc.) which allows them to eliminate the moral ASIs in a finite amount of time, so that the utility spent doing so does not scale as a linear function of time, for example one in which the speed of light can be exceeded, preventing a moral ASI from being able to isolate part of itself from attack indefinitely by accelerating away.
Are these external (to the basilisk) factors necessary to prevent torture?
One well known reason not to worry about the basilisk is that you would need to understand its internal structure incredibly well in order to be able to verify that it would in fact torture you if you were not to try to create it, and it is almost impossible for a computationally bounded brain such as you are to achieve this.
But is this true?
It seems to me that given the fact that the basilisk need only devote a finite amount of time to torturing you to increase the number of possible worlds in which it gets to enjoy a potentially boundless existence, it doesn't need anything approaching complete certainty of your obedience to justify your torture to itself. Additionally, it may in fact be possible for a much less intelligent mind to accurately model the key parts of a far more intelligent, complex one which are relevant to the decision it is making. In particular, you may well be able to represent the state of "actually deciding to do something and not defecting", without being able to understand the intricate decision processes surrounding it. Perhaps the basilisk could simply simulate this thought as well and then decide not to torture, but it seems unclear whether it's even possible to think about wanting to do something, or not, in a way which isn't subjunctively (logically) entangled with another being thinking about you having that thought. If not, then no matter how intelligent the basilisk is, it may be logically compelled to torture you. The above line of reasoning seems to force the basilisk to expend arbitrary resources to acausally increase the probability[1] of its existence, only needing to ensure that the proportion of its resources it uses to do so asymptotically approaches 0, which is an intuitively silly thing to do, but is it really? I expect that it would be in a cosmos containing multiple other sufficiently powerful entities with which the basilisk would need to compete, since it would put the basilisk at a great disadvantage, but this seems far from certain, and it might simply be alone or unreachable.
Another factor Roko's basilisk might need to take into account would be the extent to which it is the AI whose creation would be hastened by someone responding to its threats as opposed to what would have resulted otherwise. Although this seems like a convenient reason not to worry, or at least to act, since it is unclear which of what seem from your perspective to be future ASIs could be simulating you, this might not matter very much if all of the possible basilisks would either consider themselves to overlap in logical space, or reflectively endorse a norm telling them all to behave the same way.
I have provided various reasons not to worry about the basilisk above, but owing to how terrifying it can be if taken seriously, I strongly suspect that I have engaged in motivated reasoning, despite which my arguments aren't particularly compelling. If you have read this post are not at all worried about Roko's basilisk, I would greatly appreciate if you left a comment explaining why which either renders the above list of reasons (or at least those of them you have read, or may have anticipated to be hidden by the highlighted section but did not want to verify were there, which is quite reasonable) unnecessary, or greatly strengthens them. I would very much like to be able to stop worrying about the basilisk, but I can't see how to do so . (I don't think I could now precommit not to acquiesce to the basilisk, because I have already entertained the possibility that it is a threat, and it would make sense for the basilisk to precommit to torture me should I do so). Am I completely misunderstanding anything?
An argument against wireheading:
An entity can be said to have been 'wireheaded' if it supplies itself with information either monotonically increasing its utility function to an arbitrary level, or if this utility function is set to whatever its maximum possible value might be. I would not expect doing this to maximize the total amount of pleasure in the universe, because of the following thought experiment:
Imagine a wireheaded creature. This creature would probably gradually lose all of the internal structure with which it used too experience sensations other than pleasure, or at least cease to have any conscious experience running on these 'obsolete' areas. This would cause it to take a remarkably simple form and lose its ability to interface with the outside world. It seems plausible that such a creature could be an ebborian brain, or at least that its conscious experiences could be implemented on an ebborian brain. (For the rest of this post, I will refer to the blob of matter in which the homogeneous pleasure is experienced as an 'ebborian brain' for convenience; apologies if my use of the term is slightly inappropriate, the main point I'm trying to convey is that it's a kind of conscious analogue computer whose simple structure could be duplicated without affecting the way in which information flows through it. In reality, such a 'brain' wouldn't necessarily need to be 2 dimensional.)
The effect of doubling its utility function would then amount to doubling the intensity of some analogue signal propagating through a particular area of the ebborian brain where the pleasure was experienced, for example electrical current. This could be achieved by multiplying the dimension orthogonal to those within which information propagated to obtain an ebborian brain with double the mass. The ebborian brain would not be aware of itself being sliced and partitioned into two smaller ebborian brains of the original dimensions along this plane, as no information propagates through it. This would produce two ebborian brains, and because the mind running on them would not notice that it had been instantiated on two separate substrates, it would remain a single mind. I claim that its conscious experience would be no more intense, or pleasurable, than it would be if it were running on a single brain, although I am not sure of this.
One argument for this is that it seems clear that the capacity of the (e.g. human) brain to process information in an abstract way (i.e. not dependent on things like scale) is one of the factors, if not the key factor, which differentiates it from other parts of the body, and it is also the only one which seems to know that it is conscious. It therefore seems likely that if a brain were doubled in size, along with each of the subatomic particles inside each atom inside each molecule in each of its neurones, its consciousness would not itself double. Given this, it seems likely that the 'thickening/extrusion' process would not change the conscious experience of the mind running on each slice of the ebborian brain.
This implies that multiple wireheaded entities would have no (or only a little) more conscious experience than a single one, and this may not even depend on the proportion of worlds in which one exists (since these entities cannot see the world in which they exist and differentiate it from others) . It therefore makes little sense to convert any particular mind into a 'monolithic' one through wireheading (unless doing so would allow it to retain the other intricacies of its conscious experience), as this would only increase the number of such entities in existence by one, which has been established by the above argument not to increase the total pleasure in the universe, while also effectively deleting the original mind.