Well, it's one thing to explore the possibility space and completely the other one to pinpoint where you are in it. Many people will confidently say they are at X or at Y, but all that they do is propose some idea and cling to it irrationally. In aggregate, in hindsight there will be people who bonded to the right idea, quite possibly. But it's all mix Gettier cases and true negative cases.

And very often it's not even "incorrect" it's "neither correct nor incorrect". Often there is frame of reference shift such that all the questions posed before it turn out to be completely meaningless. Like "what speed?", you need more context as we know now.

And then science pinpoints where you are by actually digging into the subject matter. It's a kind of sad state of "diverse hypothesis generation" when it's a lot easier just go blind into it.

I can imagine someone several hundred years ago having figured out, purely based on first-principles reasoning, that life is no crisp category at the territory but just a lossy conceptual abstraction. I can imagine them being highly confident in this result because they've derived it for correct reasons and they've verified all the steps that got them there. And I can imagine someone else throwing their hands up and saying "I don't know what mysterious force is behind the phenomenon of life, and I'm pretty sure no one else does, either".

But is this a correct conclusion? I have an option right now to make a civilization out of brains-in-vats in a sandbox simulation similar to our reality but with clear useful distinction on life VS non life. Like, suppose there is a "mob" class.

Like, then, this person there, inside it, who figured out that life and non life is a same thing is wrong in a local useful sense, and correct in a useless global sense (like, everything is code / matter in outer reality).  People inside the simulation who found the actual working thing that is life scientifically, would laugh at them 1000 simulated years later and present it as an example of presumptuousness of philosophers. And i agree with them, it was a misapplication.

All of them, you can cook up something AIXI like in a very few bytes. But it will have to run for a very long time.

Before sleeping, I assert that the 10th digit of π equals to the number of my eyes. After falling asleep, seven coins will be flipped. Assume quantum uncertainty affects how the coins land. I survive the night only if number of my eyes equals to the 10th number of π and/or all seven coins land heads, otherwise I will be killed in my sleep.

Wil you wake up with 3 eyes?

Like, your decisions to name some digit are not equallly probable. Maybe you are the kind of person who would name 3 only if 10^12 cosmic rays hit you in precise sequence or whatever, and you name 7 with 99% prob.

AND if you are very unlikely to name the correct digit you will be unlikely to enter into this experiment at all, because you will die in majority of timelines. I.e. at t1 you decide to enter or not. At t2 experiment happens or you'll just waste time doomscrolling. At t3 you look up the digit. Your distribution at t3 is like 99% of you who chickened out.

Another possibility is Posthuman Technocapital Singularity, everything goes in the same approximate direction, there are a lot of competing agents but without sharp destabilization or power concertation, and Moloch wins. Probably wins, idk

 https://docs.osmarks.net/hypha/posthuman_technocapital_singularity

I also played the same game but with historical figure. The Schelling point is Albert Einstein by a huge margin, like 75% (19 / (19 + 6))  of them say Albert Einstein. The Schelling point figure is Albert Einstein! Schelling! Point! and no one said Schelling! 

In the first iteration of the prompt, his name was not mentioned. Then I became more and more obvious in my hints, and in the final iteration, I even bolded his name and said the prompt was the same for the other participant. And it's still Einstein!

https://i.imgur.com/XLkXTsk.png

Which means 2:1 betting odds

So, she shakes the box contemplatively. There is mechanical calendar. She knows the betting odds of it displaying "Monday" but not the credence. She thinks it's really really weird

Well, idk. My opinion here is that you bite some weird bullet, which I'm very ambivalent to. I think "now" question makes total sense and you factor it out into some separate parts from your model. 

Like, can you add to the sleeping beauty some additional decision problems including the calendar? Will it work seamlessly?

Well, now! She looks at the box and thinks there is definitely a calendar in some state. What state? What would happen if i open it?

Let's say there is an accurate mechanical calendar in the closed box in the room. She can open it but wouldn't. Should she have no expectation about like in what state this calendar is?