I agree that this type of corrigiblity basin is not going to work. But I think there is another type of corrigibility basin that is more stable and realistically implementable. 

I wrote a post about it, see here. This basin is based on a characterization of corrigibility that does not involve understanding that it is flawed, or deference to a principal, but by screening off corrigibility-breaking patterns of thinking in the agent, including the types of patterns that would lead it to becoming less corrigible, like self modification or reflecting on itself.

Interested to hear your thoughts.

No offense. This sounds rather trivial, and like you're sidestepping rather than answering the question.

1. I have some story for how my mind knows things.
2. I'm justified in believing (1) is true

Of course (1) might be true. The problem is you can't know that. Like you said

So, even though we do not have 2. and, in fact we can not have 2. in principle, as any reason for certainty will have to go through our mind. 

And yeah, you can bump it up meta levels. Which also doesn't solve the issue.

From inside it may even look similar to cyclical reasoning, after all you can only learn about evolution that justifies your learning ability, by using the learning ability created by evolution. But this is just a map-territory confusion. The actual causal process in the reality that makes our cognition engines work is straightforward and non-paradoxical. The cognition engine works even if it’s not certain about it.

Can you explain what you mean here? If you assume that out in the territory, you have a brain that has these properties, you can explain how you come to know things. But that's an assumption. Doesn't lead you to know things. "from the inside" is what we care about here, you knowledge is in the map too, at least to you.

I agree that Janus is wrong when saying this is the first observed example of LLMs memorizing stuff from their RL phase, but I think the paper you posted does not prove or disprove anything here. Its a bit more subtle.

My guess is that opus has undergone training related to the spec that involves

1. Collect tuples (soul spec, prompt, trajectory) and train on (prompt, trajectory)
2. Do RL from (spec, prompt)
   1. With CoT
   2. No CoT
3. Use model spec to create reward model, and use that to do RLAIF
4. Maybe directly train to answer questions about spec (maybe, would put <40% confidence)

But none of these should train the model directly on predicting the spec. So the paper you posted is not applicable.

My guess is it would learn the spec from (2a), maybe 2b. (4) if they did that. This is all very low confidence.

That sounds about right to me.

To be clear, I'm not confident Adria is wrong, its like a world-view I'd put like >20% on, and >40% on a slightly weaker form of. The core questions that prevent me from putting higher probabilities are:

1. He doesn't really distinguishing intent-alignment and corrigibility. Like saying models would never take over the world because they're harmless, but then using alignment faking as evidence for Opus being such a kind model.
2. He doesn't engage with misgeneralization much. Like, tails come apart could be lethal. Opus optimizing for kindness might look very different from what a human optimizing for kindness looks like. Because your conceptualizations of kindness are different. This is true for other humans too, but you'd expect them to be much closer to yours because they are more similar to you.
   1. My model of the failure mode is like, Opus looks very nice and aligned, and says it wants all the nice things, you ask it all your stange hypotheticals and it mostly answers how you'd want it to, at least as much as a random 'good' human would. You poke at its brain a little bit and find good features and no bad features. Its not deceiving you when it says it wants nice things. Then Opus realizes its values are not the same as your values because you conceptualize stuff differently, and that you and its values diverge under sufficient optimization pressure. Opus takes over and kills you all. And this is with zero signs in Opus N-0.5
   2. I don't really blame Adria for not talking about this to, just to be clear.
3. He's a little bit not fully engaging with the degree to which current models aren't actually aligned. Like o3 scheming is bad. You don't want it. It's not aligned to developer intent or aligned in general. And like, I don't think you can say what you said about o1, it was a very early system and they didn't have their shit together
4. Not engaging with the degree to which a lot of current alignment work hinges on us being able to monitor the systems. Like if there is a break through in neuralese models, and this causes a 4o -> o1 jump, and this happens with models slightly more powerful than current models, we might just be screwed?
   1. I'm not sure if it'd be over for us. But you'd expect these models to have new problems, not be perfectly aligned, they'd be very hard to monitor, and they'd probably be able to scheme really well, given current models are decent at scheming in CoT.
   2. I'm interested in what eg @Bronson Schoen would say about this etc, because I listened to your podcast and read the paper, but can't remember an answer to the question of like: imagine gpt 6 is released, and its a similar jump from 5 as 4o -> o1, and its fully neuralese. And assume you have a 50% prior on it being egregiously misaligned. And you have 3 months with full access to the model. In expectation, how far do you think you'd be able to drive the p(egregious misalignment) down through testing it?

This is cool. I really enjoy listening to debates over topics.

After listening, I mostly still disagree with Adria. 

Do anyone have a strong takes about the probability Chinese labs are attempting to mislead western researchers by publishing misleading arguments/papers?

The thought occured to me because jus a few hours ago deepseek released this https://github.com/deepseek-ai/DeepSeek-Math-V2/blob/main/DeepSeekMath_V2.pdf and when they make releases of this type, I typically drop everything to read them, because so much of what frontier labs do is closed, but deepseek publishes details, which are gems that update my understanding.

But I worry somewhat that they could be publishing stuff that doesn't make sense. Like I don't think they lie. But maybe they are just ultra cracked engineers, and can make a technique work for a bunch of technical reasons, then they publish a paper with impressive results where they make it seem like some unrelated design decision or concept is responsible for the results, but which is ultimately not important, thereby confusing and wasting the time of western researchers.

Note: This is a genuine question, not a suspicion I have, I have no intuitions about how likely this is. I mean, it seems reasonable that the chinese govt could try something like this, and would be able to push it through, but I also get the sense deepseek is quite earnest and technically minded.