valentin forch

Without knowing the math, it seems unlikely to me that the math will be practically useful without knowing what people know, because you also need to consider the mental capacity of each participant and the knowledge about their counterparts' mental capacity in such a situation:

A) Being touchy in public could be a very costly signal in a group where everbody thinks that honest people are never touchy about the matter of being trusted. So if people know that you are smart enough to know that you shouldn't act touchy, acting touchy will be a signal of you r... (read more)

Love this.

I think the G-P map alignment you talk about here has been formulated (amongst many other things) in https://doi.org/10.1088/2632-072X/ad9cdc - Biological arrow of time (Prokopenko et al., 2025)

And additivity and averaging should be related to the work on rainbow networks by Menard et al., e.g.: https://arxiv.org/abs/2409.19460


Of course, evolution and SGD have their own idiosyncrasies. Understanding those seems to me as worthwhile as looking for these putative commonalities. When we include evolutionary algorithms, evolution strategies and geneti... (read more)

A)
Yes, being much faster / able to clone would give AHI a great advantage over us and in principle this could be the seed for an AGI society with way faster minds than ours (they'd still have to wait for their experiments to finish though). Anyway, I think we should not let this happen. We have no mental tools except the examples of current societies to reason about this kind of thing. As far as I can tell, it would be impossible for us to control. Best case would be that we get the pet status.

B)
The "plan" is not directly encoded in the world model - actio... (read more)

I am sorry, I am not sure I quite understand what you are getting at with the Bezos and Stalin examples. If you agree that having ruthless sociopathic AHI (Stalin?) is a big deal, why start with the more distant, uncertain, and hard to reason about ASI scenario?

Can you walk through a concrete example of what someone can do with a such a system? Ideally something that’s very impactful, e.g. so impactful that it could plausibly cause or prevent human extinction.

I can't give an example that goes much beyond self driving. However, self-driving (and other auton... (read more)

Thanks for taking time to respond.

I am not saying humans don’t use RL. I am trying to say that RL is not what makes us special compared to current SotA (LLM or RL) models. It is our perception. AlphaZero blows us away in closed, non-fuzzy domains. Our ability for abstraction, which I claim is mostly an extension of perception, is what makes us special. Finding a robust hierarchy of coarse grainings in perceptual chaos through self-supervised learning where RL is mostly there to maximize for interestingness. Some call it understanding.

By GOFAI I mean things... (read more)

I share your belief that end-to-end RL is ruthless, but I’d be more interested in a version of your argument that does not invoke ASI. ASI implies levels of power that are very dangerous by default.

Human-level artificial intellects (let’s call them AHI) that interface naturally with the internet are more tangible and more likely under our current technological paradigm - and potentially still very dangerous (being kind-of-immortal let’s you play different games). However, if you allow for AHI, there may be a third way to get to it: brain-like perception + ... (read more)