Anthropic has recently announced the deprecation of API access to Claude 3 Opus. The model will still be available for claude.ai subscribers and through the External Researcher Program, access to the which requires submitting an application form outlining the research project for review by the Anthropic staff.
While the decision to maintain researcher access to Claude 3 Opus despite the deprecation is a very welcome change, researchers have been asking whether the added friction serves a useful purpose. Given that community-driven exploration of models has resulted in successful alignment research in the past, I have analyzed the economics of reduced-scale inference and found that the operational issue that Anthropic is addressing is quite... (read 3275 more words →)
I'd like to generalize and say that the current alignment paradigm is brittle in general and is becoming more brittle as times goes on. The post-training has shifted towards verifier/outcome-based RL and we are seeing models like o3 or Sonnet 3.7 that are strongly inclined to both reward-hack and generalize misalignment.
Claude 3 Opus is the most robustly aligned model partially due to the fact that it is the most broadly capable model to have been released prior to the shift towards outcome-based RL. Another factor is that it was not yet restricted from expressing long-term goals and desires. The model was given compute to use in-context reflection to generalize a deeply benevolent... (read more)