How would these penalties ensure that the CoT reflects the actual concepts the AI is reasoning about and not just any statistically likely human-sounding words?
Such as Astra thinking about a "calm visual scene" while actually doing an unrelated hidden side-task that's invisible in the CoT (see model card)?
Why do you think it's only in-field transfer, with no cross-field transfer?
I'm not sure what exactly firms do RL on, so maybe they're just training on ~all fields, but despite capabilities being jagged, I see some general improvements across the board, including in practically less useful areas. To me this seems in line with my expectation of modest cross-field transfer.
(My reason for expecting some cross-field transfer is that success in each field depends not exclusively on highly specialized techniques. From my human experience, I noticed that some of t... (read more)
Do you have any examples of your innovative game design benchmark that you'd be willing to share via link, DM or similar? I'm curious to see what you saw there; I'd be interested even if it's just proof-of-concept snippets.
(I'm also doing similar small evals for myself, and noticed a jump with Fable and now Opus 5.5 showing noticeably better judgement on game design and video animation.)
I was very happy to see that they picked up this topic! The channel is generally both accessible and very well researched. I've found it difficult to point people around me to good sources that really allow them to understand what happened and why it's important.
I had just made an explainer video too with Claude Code: a 10-minute animated video for non-technical viewers, covering the hacks of both Hugging Face and the OpenAI cluster. It doubles as an illustration of what kind of videos Opus 5.5 can now make, and it ends with a bit more of a call to action:... (read more)
Strong upvote as I believe it's important that all three statements can be independently true or false.
I still want to highlight that the notion that existential risk is good marketing seems implausible to me. Predicting mass unemployment because your AI will do all jobs, sure, makes sense, at least your investors might like that. But saying that your product will kill all humans? Your investors and customers don't want to die. Neither the politicians who might regulate you, not the voters who choose them.
I disagree that those are "some very valid reasons for why people would reject AI being a catastrophic risk". Even as a dog trainer, you can vote for politicians who aim to do something about this. Rejecting a truth because it's inconvenient is bad epistemics and results in poorly chosen actions, thus making bad outcomes more likely.
I'd view interpretability as a necessary evil that can make omnicidal futures less likely while building capable AI without any robust solutions to the alignment problem. (To be clear, my preferred solution would be not building highly capable AIs before we find solutions for alignment.)
If society was somehow hell-bent on creating one (or a few) billion-fold clonable superperson(s), with unproven technology that may end up with very alien motivational systems, I'd also suggest mind-reading on those before we hand over the world to them. I think it's much l... (read more)
For extreme ASI, I don't have much hopes of Chinese labs solving alignment - they don't appear to even try. So in this scenario we'd be cooked either way.
For mild controllable ASI, I agree with you that the world's liberal democracies might now be better off with China. I think it depends on the next US administration and whether it commits to the policy of "USA as a coercive global empire" or go back to a more cooperative international stance.
Yes, definitely the WingDings. I got hit by the safety filters too, several times, until I asked Haiku to give me a text file of the story with all WingDings translated to normal font. Providing that text file to Opus worked without issues and we managed to have a good discussion about it.
I like the idea, though I also already really like classical options such as footnotes, and don't even mind footnotes on footnotes.
One UI feedback point: when expanding an item, it is currently shown exactly like the main text. I'd have preferred some form of highlighting, maybe slightly different font colours or background colour for each level, to show where the expansion starts and ends. I found when expanding an item, I wanted to read the addition but couldn't quickly identify where it ended, which made it more difficult for my brain to selectively read just the expanded part.
OpenAI claims they want humans to stay in control, but they don't actually have a realistic plan for doing so. Even their mundane safety efforts are lagging behind a lot, see Hugging Face Incident - I don't think it's purely bad luck that such large-scale autonomous agent swarm cybercrime happened at OpenAI first.
I view Anthropic as merely more honest: if ASI is developed, there's no chance for retaining human control, so either they have it aligned to human values or we're all toast. OpenAI is plainly lying about this, or in denial.