valentin forch

A)
Yes, being much faster / able to clone would give AHI a great advantage over us and in principle this could be the seed for an AGI society with way faster minds than ours (they'd still have to wait for their experiments to finish though). Anyway, I think we should not let this happen. We have no mental tools except the examples of current societies to reason about this kind of thing. As far as I can tell, it would be impossible for us to control. Best case would be that we get the pet status.

B)
The "plan" is not directly encoded in the world model - actions are. The plan is either... (read more)

I am sorry, I am not sure I quite understand what you are getting at with the Bezos and Stalin examples. If you agree that having ruthless sociopathic AHI (Stalin?) is a big deal, why start with the more distant, uncertain, and hard to reason about ASI scenario?

Can you walk through a concrete example of what someone can do with a such a system? Ideally something that’s very impactful, e.g. so impactful that it could plausibly cause or prevent human extinction.

I can't give an example that goes much beyond self driving. However, self-driving (and other autonomous robotics applications) is quite a big deal (Tesla is currently worth more than "capture the light... (read 540 more words →)

Thanks for taking time to respond.

I am not saying humans don’t use RL. I am trying to say that RL is not what makes us special compared to current SotA (LLM or RL) models. It is our perception. AlphaZero blows us away in closed, non-fuzzy domains. Our ability for abstraction, which I claim is mostly an extension of perception, is what makes us special. Finding a robust hierarchy of coarse grainings in perceptual chaos through self-supervised learning where RL is mostly there to maximize for interestingness. Some call it understanding.

By GOFAI I mean things like MCTS (with good pruning heuristics), TD learning (over a hierarchy of abstract states), production systems, model predictive... (read more)

I share your belief that end-to-end RL is ruthless, but I’d be more interested in a version of your argument that does not invoke ASI. ASI implies levels of power that are very dangerous by default.

Human-level artificial intellects (let’s call them AHI) that interface naturally with the internet are more tangible and more likely under our current technological paradigm - and potentially still very dangerous (being kind-of-immortal let’s you play different games). However, if you allow for AHI, there may be a third way to get to it: brain-like perception + GOFAI. I think the human brain’s super-power does not lie in its advanced RL-algorithms to figure out our life policy. Day-to-day... (read more)