Embee's Shortform
Jan 18, 20252
Embee
61
1
18
Bostrom's argument may be underappreciated. You might like Roman Yampolskiy's work if you're deeply interested in exploring the Simulation argument.
Can you tell me your p(doom) and AGI timeline? Cause I think we can theoretically settle this:
I give you x$ now and in y years you give me back x times r $ back
Please tell me acceptable y, r for you (ofc in the sense of least-convenient-but-still-profitable)
I think we can conceivably gather data on the combination of "anthropic shadow is real & alignment is hard".
Predictions would be:
conditional on us finding alien civilizations that reached the same technological level, most of them will have been wiped by AI.
2. is my guess as to why there is a Great Filter. More so than Grabby Aliens.
That's good to know! Best of luck in your project
Feels deep but I don't get it.
Would you mind elaborating?
ANTHROPIC IMMORTALITY
Are other people here having the feeling of "we actually probably messed up AI alignment but I think we are going to survive for weird anthropic reasons"?
[Sorry if this is terrible formatting, sorry if this is bad etiquette]
I think the relevant idea here is the concept of anthropic immortality. It has been alluded to on LW more time than I could count and has even been discussed up explicitly in this context: https://alignmentforum.org/posts/rH9sXupnoR8wSmRe9/ai-safety-via-luck-2
Eliezer wrote somewhat cryptic tweets referencing it recently:
https://x.com/ESYudkowsky/status/1138936939892002816
https://x.com/ESYudkowsky/status/1866627455286648891
But for several weeks I've wished there was a definitive place on the internet where it is examined cause I have trouble wrapping my mind around the idea. Its value, theoretical defects,... (read more)
To me Feynman seems to fall quite on the von Neumann side of the spectrum.
Yes, they seem to represent two completely different types of extreme intelligence which is very interesting. I also agree that vN's ideas are more relevant for the community.
Yes. Grothendieck is undoubtedly less innovative and curious all across the board.
But I should have mentioned they are not of the same generation. vN helps build the atom bomb while G grows up in a concentration camp.
vN went along a scientific golden age. I'd argue it was probably harder to have the same impact on Science in the 1960s.
I also model G as having disdain for applying mathematical ideas to "impure" subjects. Maybe because of the Manhattan project itself as well as the escalation of the Cold War.
This would be consistent with a whole school of french mathematicians deifying pure math, N. Bourbaki in general, and being generally skeptical of the potential of pure math on the improvement of society, Roger Godement being the stereotype.
My point was that Grothendieck's mind is interesting to dissect for someone interested in a general theory of intelligence and AI alignment (and that the von Neumann metaphor becomes kinda tiring)
Pet peeve: AI community defaulted to von Neumann as being the ultimate smart human and therefore the basis of all ASI/human intelligence comparison when the mathematician Alexander Grothendieck exists somehow.
Von Neumann arguably had the highest processor-type "horsepower" we know of plus his breadth of intellectual achievements is unparalleled.
But imo Grothendieck is a better comparison point for ASI as his intelligence, while being strangely similar to LLMs in some dimensions, arguably more closely resembles what alien-like intelligence would be:
- solving "impossible" problem through meta-language and abstractions.
- able to think deeply on his own (re-discovered measure theory alone when he was a teenager, re-discovered Poincaré results when undergrad, apparently solved multiple PhD theses in... (read more)
Promising. Where can interested researchers discuss this and what does the question bank look like so far?