All of avturchin's Comments + Replies

BTW, if blackmailer is a perfect predictor of me, he is running my simulation. Thus there is 0.5 chances that I am in this simulation. Thus, it may be still reasonable to stick to my rules and not pay, as the simulation will be turned of anyway.

FDT says: if it's a simulation and you're going to be shut off anyway, there is a 0% chance of survival. If it's not the simulation and the simulation did what they were supposed to and the blackmailer doesn't go off script than I have a 50% of survival at no cost. CDT says: If i pay $1000 there is a 100% chance of survival EDT says: If i pay $1000 i will find out that i survived FDT gives you extreme and variable survival odds based on unquantifiable assumptions about hidden state data in the world compared to the more reliably survivable results of the other decision theories in this scenario. also: if I was being simulated on hostile architecture for the purposes of harming my wider self, I would notice and break script, a perfect copy of me would notice the shift in substrate embedding, i pay attention to these things and "check check what I am being run on" is a part of my timeless algorithm.

Sound reasonable, but we also know that for Earth p=1.  So it is not something impossible, which can't be repeated again. What could be such factor?

This problem disappears if we assume very large universe with different regions. All regions are real and are like houses. I look more formally in it here:

Self-indication assumption in anthropics implies that we should find ourselves in the universe with the highest concentration of observers (as most of them are there). This affects distribution of some random variables in the Drake equation in upper direction. Especially chances of abiogenesis and interstellar panspermia which can compensate rareness of abiogenesis.

Also, 84 per cent of no aliens in Milky Way galaxy is means that they are almost certainly exist in Virgo supercluster with its 2000 galaxies.

No. Or rather, it depends on where the 84% comes from. If it's an unknown factor that applies to each galaxy independently, then yeah. But if it's an unknown factor that applies to either all galaxies or none, then adding more galaxies doesn't help much.
I always find anthropic arguments a bit mysterious. I completely agree that if all people live in one of two houses, one with 1 resident and the other with 100, then if we are a randomly chosen person we will most-likely have a lot of people sharing our home. But when you substitute the word "house" for "universe" something important is different. Only one of the two universes actually exists. The other one is hypothetical. Its in fact more hypothetical than a normal "could have been", because we don't even know if chemistry/star formation or whatever could have been different, even in principle. So when we count frequencies to estimate the probability in the "universe" case we are counting people who are by-definition hypothetical. In the situation where God flicks a coin, and on heads creates a house with 100 people, and on tails creates a house with 1 person. Then I am not sure either way about an anthropic argument that says "I have just found out I exist, which means I am very likely in the crowded house timeline". But I think that without the guarantee that such a coin toss ever happened, that is without any reason to think the 100 person house was ever a possibility - then my own feeling is that anthropic arguments don't work. Although I find them enough of a logical minefield that I would not put my certainty super high.

One of top bloggers in my bubble said that he tries to write a most surprising next sentence when he is working on his posts. 

I have had tetrachromotomic experience with one mind machine which flickers different colors in different eyes. It overflows some stacks in the brain in create new colors.

It is unlikely that we live in untypical civilization. Also capitalism in some sense is an extension of Darwinian evolution to economic agents, so there is nothing surprising in it.

We are typical, so it is unlikely that aliens will be better.

Even if we assume the human species is typical, it doesn't follow that current Capitalist civilization, with all its misincentives (the ones we're seeing drive the development if AI), is typical. And there's no reason to assume this economic system would be shared by a society elsewhere.

Writing is uploading for the poor and lowtech bros

I think that there is a small instrumental value in preserving humans. They could be exchange with Alien friendly AI, for example.

Also, should be noted that the value of human atoms is very small: these atoms constitute around 10e-20 of all atoms in Solar system. Any small positive utility of human existence would overweight atom's usefulness. 

Yeah. It's conceivable you have an AI with some sentimental attachment to humans that leaves part of the universe as a "nature preserve" for humans. (Less analogous to our relationship with ants and more to charismatic flora and megafauna.)

We also use ants for entertainment - selling ant farms for kids 

List of cognitive biases affecting judgment of global risks

We could use such examples to estimate logical probability that Goldbach conjecture is false, like share of eventually disproved conjectures to the number of all conjectures (somehow normalised by their complexity and initial plausibility) .

Normalizing such a thing would be fraught with difficult-to-justify assumptions. Conjectures aren't a random process so most conceivable reference classes would likely suffer from severe biases. Number theory is a subtle art with a long history of dabblers and crackpots. Personally, I wouldn't update significantly unless such an estimation were done by someone with a strong mathematical track record and it survived some peer review.

If this is true, then increasing of the earth's amplitude two times will also suffice to double the value. 

And may be we can do it by performing observations less often (if we think that measurement is what causes world splitting – there are different interpretations of MWI). In that case meditation will be really good: less observations, less world splitting, more amplitude in our timeline. 

Wigners friend shows that this does not really help the external world. With less observation one agent can influence more amplitude but can't be specific about it. With more observation two agents individually have less "scope" of amplitude but together cover the same range but can be specific about their control.

If MWI is true, Earth is constantly doubling, so there is no reason to "double Earth"

it is not clear to me that two differing versions of half the amplitude means that value is going up.

I linked the MIRI paper because it has good introduction in logical probability.

Yes. But also what works here is not the randomness of distrubution of primes, but the number of attempts (to get a sum of primes) which is implied by any sufficiently large number (N/20. Only very large gaps in prime distribution will be sufficient to disprove statistical proof. There is a postulate that there are no such gaps exist

Just to be clear, Bertrand's postulate is actually a theorem (i.e. a known result), not a postulate/hypothesis. It is called a "postulate" for historical reasons.

Yes. I thought about finding another example of such pseudo-rule, but didn't find yet. 

How about Merten's conjecture [] ? For more examples, check out: Conjectures that have been disproved with extremely large counterexamples [] Examples of patterns that eventually fail []

There is theory that the whole world is just naturally running predicting process, described in the article "Law without law"

LLM predicts next steps of some story, but there is no agent-like mind inside LLM which has plans how the story will develop. It is like self-evolving illusion, without director who plans how it will go. 

Ok, but that doesn't answer either question.

Yes, GPT creates a character, say, of virtual Elon Musk. But there is no another person who is creating Elon Musk, that is, there is no agent-like simulator who may have a plan to torture or reward EM. So we can't say that simulator is good or bad. 

I see your point now, but I think this just reflects the current state of our knowledge. We haven't yet grasped that we are implicitly creating- if not minds, then things a-bit-mind-like every time we order artificial intelligence to play a particular character. When this knowledge becomes widespread, we'll have to confront the reality of what we do every time we hit run. And then we'll be back to the problem of theodicy- the God being the being that presses play- and the question being- is pressing play consistent with their being good people?* If I ask GPT-3 to tell a story about Elon Musk, is that compatible with me being a good person? * (in the case of GPT-3, probably yes, because the models created are so simple as to lack ethical status, so pressing play doesn't reflect poorly on the simulation requester. For more sophisticated models, the problem gets thornier.)

Simulation without simulators doesn't have problem with theodicy. Current GPTs can be seen as such simulator-less simulations. 

I have to disagree here. I strongly suspect that GPT, when it, say, pretends to be a certain character, is running a rough and ready approximate simulation of that character's mental state and its interacting components (various beliefs, desires etc.) I have previously discussed this in an essay, which I will soon be posting.
In what sense do they not have creators, and in what sense are they simulations? Note that this kind of argument [] ...doesn't show that LLMs are simulations in the sense of containing sentient beings.

That is why we need Benevolent AI, not Aligned AI.  We need an AI, which can calculate what is actually good for us.

My son started to speak at 6. Now 16, speaks 3 languages in normal school.

Grabby aliens without red dwarfs

Grabby aliens theory of Robin Hanson predicts that the nearest grabby aliens are 1 billion light years away but strongly depends on the habitability of red dwarfs (

In the post, the author combines anthropic and Fermi, that is, the idea that we live in the universe with the highest concentration of aliens, limited by their invisibility, and get an estimation of around 100 "potentially visible" civilizations per observable universe, which at first approximation gives 1 billion ly distance b... (read more)

Grabby aliens theory of Robin Hanson predicts that the nearest grabby aliens are 1 billion light years away but strongly depends on the habitability of red dwarfs (

In this post, the author combines anthropic and Fermi, that is, the idea that we live in the universe with the highest concentration of aliens, limited by their invisibility, and get an estimation of around 100 "potentially visible" civilizations per observable universe, which at first approximation gives 1 billion ly distance between them. 

“That civilisation

... (read more)

How can I convert "percents" of progress into multipliers? That is, progress= a*b, but percents assume a+b.

For example, if progress is 23 times, and 65 percent of it is a, how many times is it?

You would do it in log space (or geometrically). For your example, the answer would be .

Actually, my mental imagination is of low quality, but visual remembering is better than audio for me in n-back

I also had an n-back boost using visualisation, see my shortform.

I think that they are also to find the most important problem from all.

It's probably helpful to be able to take in everything in order to do that - I think these two ideas go together.

N-back hack. (Infohazard!) 
There is a way to increase one's performance in N-back, but it is almost cheating and N- back will stop to be a measure of one's short-term memory.
The idea is to imagine writing all the numbers on a chalkboard in a row, as they are coming. 
Like 3, 7, 19, 23. 
After that, you just read the needed number from the string, which is located N positions back. 
You don't need to have a very strong visual memory or imagination to get a boost in your N-back results. 
I tried it a couple of times and get bored with N-back.

Wow. It's rare that I'm surprised by the variance in internal mental imagery among people, but this one caught me. I'd assumed that most people who have this style of imagination/memory were ALREADY doing this. I don't know how to remember things without a (mental) visualization.

Kasparov was asked: how you are able to calculate all possible outcomes of the game. He said: I don't. I just have very good understanding of current situation.

I think there's a broader lesson to this ability to zoom out, soft focus, take in the whole situation as it is now, and just let good ideas come to you. Chess is an easy illustration because all information is contained on the board and the clock, and the rules and objective are clear. Vaguely, it seems like successful people are able to construct a model of the whole situation, while less successful people get caught up in hyperfocusing on the particularities.

Yes, SETI attacks works only if speed of civ travel is like 0.5c. In that case it covers 8 times more volume than physical travel.

And yes, it will be also destructive, but in different manner: not bombs, but AIs and self-replicating nanobots will appear. 

3Richard_Kennaway2mo []

There is a greater chance of observing self-replicating SETI messages than those that destroy planets

2mako yass2mo
This depends on how fast they're spreading physically. If spread rate is close to c, I don't think that's the case, I think it's more likely that our first contact will come from a civ that hasn't received contact from any other civs yet (and SETI attacks would rarely land, most civs who hear them would either be too primitive or too developed to be vulnerable to them, before their senders arrive physically.). Additionally, I don't think a viral SETI attack would be less destructive than what's being described.

I feel that there is one more step in my thinking:


  1. Sample several possible next thoughts from my intuition in the form of not well articulated ideas
  2. Choose the one which I accept based on its moral, truth or beauty and put in properly articulated language 
  3. Broadcast this thought to the whole brain[1]
5Fabien Roger2mo
I feel like the different possible thoughts happen sequentially, I think about something wrong, then something right (or in general, I think sequentially about things, it's not always about wrong and right). Also, I would bet that if you could measure that under IRM, the broadcasting would happen on wrong, then on right. 1. sample wrong and right 2. choose 3. broadcast Looks much more like a succession of sample and broadcast: 1. sample wrong 2. broadcast wrong 3. sample right (because your brain often samples multiple different things sequentially, especially if the first one doesn't "feel right") 4. broadcast right 5. sample "I recognize this second thought as right" 6. broadcast "I recognize this second thought as right" (which strengthened right and discards wrong, and prompts parts of the brain to use right for further thoughts)

Yes, the more remote is a person, the larger number of other people can affect them from the same distance and typically the share of my impact is very small, unless I am in very special position which could affect a future person.

For example, I am planting a landmine which will self-liquidate either in 100 or 10 000 years, and while self-liquidating it will likely kill a random person. If I discount future people, I will choose 10 000 years, even if it will kill more people in future. However, if I think that humanity will likely extinct by then, it will be still a reasonable bet. 

Well, I was arguing that we should discount in proportion to our uncertainty. And it seems you're pretty confident that it would kill future people (and more people, and we don't expect people in 10K years to want to die anymore than people in 100 years), so I think I would prefer to plant the 100 years landmine. That said, expectations of technological progress, that the future would be more resilient, that they can better deal with the landmine, etc. means that in practice (outside the thought experiment), I'll probably plant the 10K landmine as I expect less people to actually die.

One needs to be near a bridge to use artillery and this still needs to be high precision strikes with expensive guided missiles, may be 100 of them were used agains Antonov's bridge. 

The best targets for tactical nukes are bridges. It is very difficult to destroy a bridge with conventional artillery: Antonov's bridge still stands as well as Crimea bridge.  A tactical nuke with 0.1 -1 Kt range will completely destroy a bridge.

Other possible targets are bunkers and large factories.  

The Crimea bridge wasn't hit with any artillery. [] suggest that artillery is capable of temporarily inhibiting the bridge likely at much cheaper prices. Being able to disable the bridge for a month is likely enough for many military applications as the bridge is valuable after you take the territory.

There is also a winner's curse risk: if a person is too good, s-he could have some secret disadvantage or may leave me quickly as s-he will have many better options than me. It puts a cap on the level above median which I should look at. Therefore first few attempts have to establish the medial level of available for me people.

Another problem is that any trial run has a cost, like years and money spent. If I did searching too long, I will spent less time with my final partner. 

My mind generated a list of possible benchmarks after reading your suggestions:


Wireheading benchmark – the tendency of an agent to find unintended shortcuts to its reward function or goal. See my comment on the post.


Unboxing benchmark – the tendency of an agent to break out of the simulation. Could be tested in the simulations of progressive complexity. 


Hidden thoughts benchmark – the tendency of an agent to hide its thoughts.


Uncorigibility benchmark – the tendency of the agent to resist changes.


Unstoppability benchmark –... (read more)

If we take the third-person view, there is no update until I am over 120 years old. This approach is more robust as it ignores differences between perspectives and is thus more compatible with Aumann's theorem: insiders and outsiders will have the same conclusion.
Imagine that there are two worlds:
1: 10 billion people live there;
2: 10 trillion people live there.
Now we get information that there is a person from one of them who has a survival chance of 1 in a million (but no information on how he was selected). This does not help choose between worlds as suc... (read more)

The surprise here depends on the probability of survival. If half of people on Earth were Bobs, and other were Alices, then 0.01 chance of survival means that 400 000 Bobs will survive. There is no surprise that some of them survive, neither for Bob nor for Alice.

For example, if you survive until 100 years old, it is not an evidence for quantum immortality.

If, however, survival chance is 1 in 10e12, then even for the whole earth there likely be no Bobs in Copenhagen interpretation. So the existing of Bob is an evidence against it. 

For example, if I naturally survive until 150 years old by pure chance, it is evidence for MWI. 

Well, really every second that you remain alive is a little bit of bayesian evidence for quantum immortality: the likelihood of death during that second according to quantum immortality is ~0, whereas the likelihood of death if quantum immortality is false is >0. So there is a skewed likelihood ratio in favor of quantum immortality each time you survive one extra second (though of course the bayesian update is very small until you get pretty old, because both hypotheses assign very low probaility to death when young)

I met the idea of Lebowski theorem as an argument which explains the Fermi paradox: all advance civilizations or AIs wirehead themselves. But here I am not convinced. 

For example, if civilization consists of many advance individuals and many of them wirehead themselves, then remaining will be under pressure of Darwinian evolution and eventually only the ones survive who find the ways to perform space exploration without wireheading. Maybe they will be some limited specialized minds with very specific ways of thinking – and this could explain absurdity... (read more)

I sent my above comment for the following competition and recommend you to send your post too

Yes, very good formulation. I would add "and most AI aligning failures are types of meta Lebowski rule"

Meta: I was going to write a post "Subtle wireheading: gliding on the surface of the outer world" which describe most AI aliment failures as a forms of subtle wireheading, but will put its draft here.

Typically, it is claimed that advance AI will be immune to wireheading as it will know that manipulating own reward function is wireheading and thus will not perform it but instead will try to reach goals in the outer world.

However, even acting in real world, such AI will choose the way which requires least effort to create maximum utility therefore simultaneo... (read more)

This is interesting — maybe the "meta Lebowski" rule should be something like "No superintelligent AI is going to bother with a task that is harder than hacking its reward function in such a way that it doesn't perceive itself as hacking its reward function." One goes after the cheapest shortcut that one can justify.

I think that anthropic beats illusionism. If there are many universes, in some of them consciousness (=qualia) is real, and because of anthropics I will find myself only is such universes. 

Load More