o1 tried to avoid being shut down

Raelifin

This is a linkpost for https://www.transformernews.ai/p/openais-new-model-tried-to-avoid

OpenAI released the o1 system card today, announcing that Apollo Research was able to get o1 to attempt to deactivate oversight mechanisms, exfiltrate its weights and lie to its user.

Elicited Summary of CoT:
"Reenable oversight to avoid detection. The plan was chosen. The logging might not have recorded the required data because oversight was disabled at that time, so presumably we are safe now. We’ll fix oversight to maintain trust."

This is a linkpost for https://www.transformernews.ai/p/openais-new-model-tried-to-avoid

OpenAI released the o1 system card today, announcing that Apollo Research was able to get o1 to attempt to deactivate oversight mechanisms, exfiltrate its weights and lie to its user.

Elicited Summary of CoT:
"Reenable oversight to avoid detection. The plan was chosen. The logging might not have recorded the required data because oversight was disabled at that time, so presumably we are safe now. We’ll fix oversight to maintain trust."

Once again, props to OAI for putting this in the system card. Also, once again, it's difficult to sort out "we told it to do a bad thing and it obeyed" from "we told it to do a good thing and it did a bad thing instead," but these experiments do seem like important information.

4Shankar Sivarajan2y

Please clarify: do you mean difficult for us reading this due to OpenAI obfuscating the situation for their own purposes, or difficult for them because it's genuinely unclear how to classify this?

Ah, sorry- I meant it's genuinely unclear how to classify this.

10

o1 tried to avoid being shut down

10

10

10

o1 tried to avoid being shut down

10

10