I agree it's possible and it's worth thinking through considerations like this. But I still don't think this is a good model of journalists' incentives.
In practice, "probability of being seen as inaccurate" is the term that dominates, which means inaccuracies tend to show up at points in the news article that face the least scrutiny, eg the part of an AI article where the journalist rushes through what a transformer is. These are the parts that are often least important to readers, and least important to you as a source.
And then I would describe the motivation more as "career success" than "political benefit". As in getting a big scoop or writing a successful story, more than pushing a particular agenda. I think what journalists' consider a successful story is kind of correlated with importance to the reader, barely correlated with what's impactful, and barely correlated with how frustrating it would be for you to be misquoted. Consider the ChatGPT suicide example: the journalist is focused on their big scoop, but probably cares much less about the paragraph I pulled out. Ditto for readers. But I think it's very valuable it was included.
I'll have more on this in the epistemics post.
1To make the analogy stronger, what if it only inverts it 1/10 times? Then I think the answer is non-obvious and depends on your principles.
If you're deontological about it, I think you could make a case that your hands are not dirty for making the best of a bad system.
If you're consequentialist about it, I'm saying the 9/10 accuracies could outweigh the 1/10 inaccuracies. And as Zack said, the 1/10 errors are rarely true inversions. That's why
even if you do get misquoted, it doesn't mean talking to the journalist was net-negative, even for that particular piece and even ex-post. As annoying as it is, it might be outweighed by the value of steering the article in positive ways.
1You are doing the lord's work fr
MIT Tech Review doesn't break much news. Try Techmeme.
Re "what people are talking about"
Sure, the news is biased toward topics people already think are important because you need readers to click etc etc. But you are people, so you might also think that at least some of those topics are important. Even if the overall news is mostly uncorrelated with your interests, you can filter aggressively.
Re "what they're saying about it"
I think you have in mind articles that are mostly commentary, analysis, opinion. News in the sense I mean it here tells you about some event, action, deal, trend, etc that wasn't previously public. News articles might also tell you what some experts are saying about it, but my recommendation is just to get the object-level scoop from the headline and move on.
Re is it worth the time of sifting through
Skimming headlines is fast. Maybe the news tends to be less action-relevant for your research, but I bet AI safety collectively wastes time and misses out on establishing expertise by being behind the news. Reading Zvi's newsletter falls under what I'm advocating for (even though it's mostly that what-people-are-saying commentary, the object-level news still comes through.)
Conditioning as a Crux Finding Device
Say you disagree with someone, e.g. they have low pdoom and you have high pdoom. You might be interested in finding cruxes with them.
You can keep imagining narrower and narrower scenarios in which your beliefs still diverge. Then you can back out properties of the final scenario to identify cruxes.
For example, you start by conditioning on AGI being achieved - both of your pdooms tick up a bit. Then you also condition on that AGI being misaligned, and again your pdooms increase a bit (if the beliefs move in opposite directions, that might be worth exploring!). Then you condition on the AGI self-exfiltrating, and your pdooms nudge up again.
Now you've found a very narrow scenario in which you still disagree! You think it's obvious that a misaligned AGI proliferating around the world is an endgame, they don't see what the big deal is. From there, you're in a good position to find cruxes.
(Note that you're not necessarily finding the condition of maximum disagreement, you're just trying to get information about where you disagree.)
Got it thanks!
(eg. any o1 session which finally stumbles into the right answer can be refined to drop the dead ends and produce a clean transcript to train a more refined intuition)
Do we have evidence that this is what's going on? My understanding is that distilling from CoT is very sensitive—reordering the reasoning, or even pulling out the successful reasoning, causes the student to be unable to learn from it.
I agree o1 creates training data, but that might just be high quality pre-training data for GPT-5.
Why does it make the CoT less faithful?
Favorite post of the year so far!
Re going along with lies - Yeah, I think the coverage of data center water usage has been an example of that at its worst :/
Re journalists sitting on scoops - I'm curious if you're able to share any examples? I don't doubt that it happens.