TLDR: Just pick up the phone. Agree on permissions for information before you share it (see Step 4). I have worked as a professional journalist covering AI for over a year, and during that time, multiple people working in AI safety have asked me for advice on engaging with journalists....
Sources' motivations for talking to journalists are a bit of a puzzle. On the one hand, it's helpful for journalists to work out what those motivations are, to keep sources invested in the relationship. On the other hand, sources behave in perplexing ways, for instance sharing information against their own...
TLDR: Instead of scrutinizing minor errors, ask what process generated the text in front of you, and then update accordingly. Focus on headlines and source attributions. If you work in AI safety, I think you should probably read a bit more news than you do now (though giving advice like...
> Nobody said it was easy > No one ever said it would be this hard > > — Coldplay, The Scientist In the wake of Dylan Matthews’ recent piece on the cost effectiveness of rebuilding Notre Dame, I have seen many people on Twitter hand wave away the demandingness...
Summary This winter, MATS will be running our seventh program. In early-mid 2024, 46% of alumni from our first four programs (Winter 2021-22 to Summer 2023) completed a survey about their career progress since participating in MATS. This report presents key findings from the responses of these 72 alumni. *...
That's right: a neartermist take! Cower before its sublime wrath! The Resurrection of the Author > Here, hold my bee. > > — Someone with beauty in their eye There are many theories of aesthetics that seek to explain the value of art and the nature of beauty. On some...
Co-Authors: @Rocket, @LauraVaughan, @McKennaFitzgerald, @Christian Smith, @Juan Gil, @Henry Sleight, @Matthew Wearden, @Ryan Kidd The ML Alignment & Theory Scholars program (MATS) is an education and research mentorship program for researchers entering the field of AI safety. This winter, we held the fifth iteration of the MATS program, in which...
Thanks! :)
Re going along with lies - Yeah, I think the coverage of data center water usage has been an example of that at its worst :/
Re journalists sitting on scoops - I'm curious if you're able to share any examples? I don't doubt that it happens.
I agree it's possible and it's worth thinking through considerations like this. But I still don't think this is a good model of journalists' incentives.
In practice, "probability of being seen as inaccurate" is the term that dominates, which means inaccuracies tend to show up at points in the news article that face the least scrutiny, eg the part of an AI article where the journalist rushes through what a transformer is. These are the parts that are often least important to readers, and least important to you as a source.
And then I would describe the motivation more as "career success" than "political benefit". As in getting a big scoop or writing a... (read more)
To make the analogy stronger, what if it only inverts it 1/10 times? Then I think the answer is non-obvious and depends on your principles.
If you're deontological about it, I think you could make a case that your hands are not dirty for making the best of a bad system.
If you're consequentialist about it, I'm saying the 9/10 accuracies could outweigh the 1/10 inaccuracies. And as Zack said, the 1/10 errors are rarely true inversions. That's why
even if you do get misquoted, it doesn't mean talking to the journalist was net-negative, even for that particular piece and even ex-post. As annoying as it is, it might be outweighed by the value of steering the article in positive ways.
You are doing the lord's work fr
TLDR: Just pick up the phone. Agree on permissions for information before you share it (see Step 4).
I have worked as a professional journalist covering AI for over a year, and during that time, multiple people working in AI safety have asked me for advice on engaging with journalists. At this point, I've converged on some core lessons, so I figured I should share them more widely.
I've also reviewed some of the prior art about how to talk to journalists on LessWrong and found it unsatisfying. The answer to the question often seems to be "Don't."[1]
Unsurprisingly then, I think many people feel like they are not prepared to talk to journalists. They... (read 3082 more words →)
Sources' motivations for talking to journalists are a bit of a puzzle. On the one hand, it's helpful for journalists to work out what those motivations are, to keep sources invested in the relationship. On the other hand, sources behave in perplexing ways, for instance sharing information against their own interests, so it's often best to treat their psychology as unknowable.
Reflecting on sources' willingness to share compromising information, one mystified AI journalist told me last weekend, "no reasonable person would do this."
But to the extent I can divine their motivations, here are some reasons I think people talk to me at work:
MIT Tech Review doesn't break much news. Try Techmeme.
Re "what people are talking about"
Sure, the news is biased toward topics people already think are important because you need readers to click etc etc. But you are people, so you might also think that at least some of those topics are important. Even if the overall news is mostly uncorrelated with your interests, you can filter aggressively.
Re "what they're saying about it"
I think you have in mind articles that are mostly commentary, analysis, opinion. News in the sense I mean it here tells you about some event, action, deal, trend, etc that wasn't previously public. News articles might also tell you what some... (read more)
TLDR: Instead of scrutinizing minor errors, ask what process generated the text in front of you, and then update accordingly. Focus on headlines and source attributions.
If you work in AI safety, I think you should probably read a bit more news than you do now (though giving advice like this is perilous because surely some of you are news junkies who need to hear the opposite message).[1]
In particular, I think you should subscribe to Techmeme and skim the headlines in the daily news roundup. Techmeme is a news aggregator that will collate the biggest tech stories of the day in a convenient email, and—crucially—they write their own headlines to tell you what's... (read 1367 more words →)
Conditioning as a Crux Finding Device
Say you disagree with someone, e.g. they have low pdoom and you have high pdoom. You might be interested in finding cruxes with them.
You can keep imagining narrower and narrower scenarios in which your beliefs still diverge. Then you can back out properties of the final scenario to identify cruxes.
For example, you start by conditioning on AGI being achieved - both of your pdooms tick up a bit. Then you also condition on that AGI being misaligned, and again your pdooms increase a bit (if the beliefs move in opposite directions, that might be worth exploring!). Then you condition on the AGI self-exfiltrating, and your pdooms nudge... (read more)
Got it thanks!
(eg. any o1 session which finally stumbles into the right answer can be refined to drop the dead ends and produce a clean transcript to train a more refined intuition)
Do we have evidence that this is what's going on? My understanding is that distilling from CoT is very sensitive—reordering the reasoning, or even pulling out the successful reasoning, causes the student to be unable to learn from it.
I agree o1 creates training data, but that might just be high quality pre-training data for GPT-5.
Nobody said it was easy
No one ever said it would be this hard— Coldplay, The Scientist
In the wake of Dylan Matthews’ recent piece on the cost effectiveness of rebuilding Notre Dame, I have seen many people on Twitter hand wave away the demandingness objection to utilitarianism,[1] which says that always trying to do the most good is an impossible obligation.
To redeem their version of morality from the demangingness objection, the tweeters assert that some good deeds are supererogatory, which is philosophy for “nice to do, but not obligatory.” The problem is that they do not present a reason why doing more good would ever be supererogatory, other than the implicit convenience of ducking... (read more)
This winter, MATS will be running our seventh program. In early-mid 2024, 46% of alumni from our first four programs (Winter 2021-22 to Summer 2023) completed a survey about their career progress since participating in MATS. This report presents key findings from the responses of these 72 alumni.
That's right: a neartermist take! Cower before its sublime wrath!
Here, hold my bee.
— Someone with beauty in their eye
There are many theories of aesthetics that seek to explain the value of art and the nature of beauty. On some of these theories, the artist's role is subordinate to the viewer's. Aesthetic value is located in a private experience between the viewer and the artistic object. The beauty of an intricate painting is not too different from the beauty of a sunset: the painter is irrelevant.
If you think of art in these terms, you might be excited by the prospect of a proliferation of cheap and beautiful AI-generated artwork.... (read 2998 more words →)
I would really like to see a post from someone in AI policy on "Grading Possible Comprehensive AI Legislation." The post would lay out what kind of safety stipulations would earn a bill an "A-" vs a "B+", for example.
I'm imagining a situation where, in the next couple years, a big omnibus AI bill gets passed that contains some safety-relevant components. I don't want to be left wondering "did the safety lobby get everything it asked for, or did it get shafted?" and trying to construct an answer ex-post.
File under 'noticing the start of an exponential': A.I. Helped to Find a Vast Source of the Copper That A.I. Needs to Thrive
Scott Alexander says:
Suppose I notice I am a human on Earth in America. I consider two hypotheses. One is that everything is as it seems. The other is that there is a vast conspiracy to hide the fact that America is much bigger than I think - it actually contains one trillion trillion people. It seems like SIA should prefer the conspiracy theory (if the conspiracy is too implausible, just increase the posited number of people until it cancels out).
I am often confused by the kind of reasoning at play in the text I bolded. Maybe someone can help sort me out. As I increase the number of people in the conspiracy... (read more)
Co-Authors: @Rocket, @LauraVaughan, @McKennaFitzgerald, @Christian Smith, @Juan Gil, @Henry Sleight, @Matthew Wearden, @Ryan Kidd
The ML Alignment & Theory Scholars program (MATS) is an education and research mentorship program for researchers entering the field of AI safety. This winter, we held the fifth iteration of the MATS program, in which 63 scholars received mentorship from 20 research mentors. In this post, we motivate and explain the elements of the program, evaluate our impact, and identify areas for improving future programs.
Key details about the Winter Program:
The Assignment:
(4 hours) Write an Economist-style explainer article on how LLMs work. You’ve just started as an AI reporter at The Economist, and your editor’s realised there’s no good Economist Explains style piece on how LLMs work. They’ve asked you to write one. It should be 500 words, and in the style of other Economist Explains pieces.
Examples: Economist explainer on biological weapons; Economist explainer on diffusion models; FT explainer on transformers.
Thank you to Shakeel Hashim for feedback! Shakeel previously worked at The Economist as an editor.
Since OpenAI released ChatGPT in November 2022, large language models (LLMs) have gained international attention. A language model is a piece of AI software designed for tasks... (read 595 more words →)
I just came across this word from John Koenig's Dictionary of Obscure Sorrows, that nicely capture the thesis of All Debates Are Bravery Debates.
redesis n. a feeling of queasiness while offering someone advice, knowing they might well face a totally different set of constraints and capabilities, any of which might propel them to a wildly different outcome—which makes you wonder if all of your hard-earned wisdom's fundamentally nonstraferable, like handing someone a gift card in your name that probably expired years ago.
... (read 2375 more words →)“A great deal of thinking well is about knowing when to think at what levels of abstraction. On the one hand, abstractions drop detail, and the reliability of inferences using them varies in complex ways with context. On the other hand, reasoning abstractly can be much faster and quicker, and can help us transfer understanding from better to less known cases via analogy . . . my best one factor theory to explain the worlds I’ve liked best, such as “rationalists”, is that folks there have an unusually high taste for abstraction . . . Thus my strongest advice for my fellow-traveler worlds of non-academic conversation is: vet your abstractions more. For example,
Co-Authors: @Rocket, @Juan Gil, @Christian Smith, @McKennaFitzgerald, @LauraVaughan, @Ryan Kidd
The ML Alignment & Theory Scholars program (MATS, formerly SERI MATS) is an education and research mentorship program for emerging AI safety researchers. This summer, we held the fourth iteration of the MATS program, in which 60 scholars received mentorship from 15 research mentors. In this post, we explain the elements of the program, lay out some of the thinking behind them, and evaluate our impact.
Key details about the Summer 2023 Program:
Today I am thankful that Bayes' Rule is unintuitive.
Much ink has been spilled complaining that Bayes' Rule can yield surprising results. As anyone who has taken an introductory statistics class knows, it is difficult to solve a problem that requires an application of Bayes' Rule without plugging values into the formula, at least for a beginner. Eventually, the student of Bayes may gain an intuition for the Rule (perhaps in odds form), but at that point they can be trusted to wield their intuition responsibly because it was won through disciplined practice.
This unintuitiveness is a feature, not a bug because it discourages motivated reasoning. If Bayes' Rule were more intuitive, it would... (read more)
AGI Should Have Been a Dirty Word
Epistemic status: passing thought.
It is absolutely crazy that Mark Zuckerberg can say that smart glasses will unlock personal superintelligence or whatever incoherent nonsense and be taken seriously. That reflects poorly on AI safety's comms capacities.
Bostrom's book should have laid claim to superintelligence! It came out early enough that it should have been able to plant its flag and set the connotations of the term. It should have made it so Zuckerberg could not throw around the word so casually.
I would go further, and say that the early safety writing on AGI should have been enough that the labs were too scared to say in public in... (read more)