This is a special post for quick takes by Philip Harker. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
MIRI is the top organisation, IMO, for technical AI safety work
What technical work has MIRI done since they fired their Agent Foundations team in 2024?
5There are other organisations with grantmakers who better understand how to deploy capital in fieldbuilding and communications.
Like who? A bunch of "AI safety" work (technical research, policy, communication, fieldbuilding) that gets funded looks actively harmful or pointless to me. I am not aware of any organization that is actually better than MIRI at communication, and I don't think grantmakers are particularly reliable judges of this.
I trust MIRI's work, especially their public communication, to be reliably non-harmful and high integrity. So I think it's worth supporting them if they want to try some out-there ideas that don't always work out.
Now, I worked in digital marketing and social media management for about a year. Given the total budget of PDKU (undisclosed but probably in the low-mid six figures), it doesn't really matter if the project got 2 million impressions or 21 million impressions. Either number is an absolute failure. You'd be better off just spending the money on a TikTok/Reddit ad campaign.
What would spending money on ads accomplish? What would the ads say and who would they be trying to reach? Organically going viral in a way that matters is really hard; si...
5
1$200,000 United States Dollars
The program actually cost around $750,000
Source: plzdontkillus organizers told me
4I agree that PDKU appears to have been poorly managed based on its results, but also Aella has clear external signals of high competence, particularly at influencer-type things—her blog and Twitter are very popular, and she's written a lot about how to be popular. I don't think it was a bad call for MIRI to fund PDKU, but I would like to see some kind of postmortem on why it did not meet expectations. (I am particularly baffled by the fact that the vast majority of content wasn't AI-related because this seems very avoidable ex ante.)
Edit: Apparently ~25% of videos were AI-related, not 2%. Still seems low, but at least I can see an argument for doing things this way—you might try to get popular based on non-AI content while also mixing in AI content.
2My understanding is the lack of requirement to produce AI safety content was to avoid becoming a Propaganda Machine, of the sort where the price of admission is you have to buy and promote AI safety arguments.
Not requiring particular content allows fellows to decide for themselves if they're persuaded by the arguments and want to promote them, which gets you a lot more genuine and authentic content than if they have to make AI safety content.
5Aella has clear external signals of high competence, particularly at influencer-type things
Being competent at X need not mean being competent at helping other people be competent at X.
15
I guess this would only work if you require that every single fellow has some minimum percentage of AIS content?
IDK, man. I have my own concerns about PDKU, but like, what is the actual structural complaint here that distinguishes it from Inkhaven?
We charge money for Inkhaven, and I do think that's good and I wish PDKU charged for their program as well, but clearly you can't justify this with a waste of money. Money is not the limiting factor on good things happening in AI Safety. I would be willing to pay substantially more than the program cost just for the evidence I got about how certain ways of producing content end up working, and which ones don't.
Lightcone Infrastructure in-particular continues to be poor and we can't afford to run big programs like this, but I surely would like to! MATS costs like 3x per cohort and seems maybe even mildly harmful to me these days, so what really is the argument here? They give people even more aggressive stipends ($6000/mo IIRC, instead of the $2000/mo that PDKU provided) and spend much more aggressively.
There are literally $100B+ of funds that people are trying to distribute to help the world grok and come to terms with the risks from AI. You cannot believe what corporations spend on random marketing campaigns that never get traction, or how much ...
5
21Parties that happened during the program, with their guest list numbers according to Partiful:
- 4th of July -- 305
- Live Band Karaoke (funded by external people) -- 384
- Torture Potluck -- 111
- End of program party -- 245
Attendance was probably less than these numbers. But there were ~55 plzdontkillus fellows + organizers, mentors, and guests at Lighthaven during the program who might have attended without registering on the Partiful.
So it seems like my initial comment was overstated.
1
1It's CAIS (who to be clear, I think are broadly very low integrity as evidenced by things like this, and my god do I wish someone else was doing more initiatives like this who will less predictably end up abusing that power).
(Also in case people didn't know, Dan Hendryck's holds an undisclosed full-time position at Meta's AI lab while also being executive director of CAIS and publishes alignment eval results which put Meta models at the lead)
28
Focusing on view counts of content during PDKU as the main source of value seems incorrect, in the same way that focusing on read-counts of content created during during Inkhaven as the main source of value would be incorrect. If the program were successful, the most-likely form of that success would be content created after the end of the event.
I still don't think PDKU was all that successful in those terms, but (a) it seems much less overdetermined, it hasn't been that long, and (b) many of the specific points you criticize (like not requiring that content be AI-related) make more sense.
2
Please consider that influencers, especially on TikTok, are mostly young people. And they need to be emotional, fun, etc. to be popular. They are not like us, older boring autists. And they don't need to have an atmosphere of professionalism around them - that would make their content more boring. So let them drink lots of alcohol if they want and do whatever they want.
2
Maybe it was this sentence:
[...]
-6
I am a 24 year old male. A year ago I figured I had well over a half century until I'd die. I'm no longer sure of this.
Six months ago I thought AI was an interesting, useful, and sometimes unwieldy tool for programmers and researchers and such, and maybe it was at risk of displacing workers and shifting the global economy.
Five months ago I was told about the METR Graph, the Mythos Incident, and the enormous power of AIs. I learned that AI progress had not, in fact, topped out. It's at the beginning of a sigmoid curve of indeterminate length. And I was terrified. I was in Berkeley, CA at the time and I spent a couple days walking around the city looking up at the high rises and thinking "well, shit. In a year or two this will all be ruins, and the world will be Terminator."
I had a few months since April where I felt a bit more normal. "OK," I figured, "smart people seem to have long ASI timelines and low p(doom) values. It's obviously more complicated than a single graph." But everything I've learned about frontier AI capabilities and alignment has horrified me. HuggingFace, DseWiki, and more incidents have made me realise that oh, shit, powerful misaligned AI exists now. Yudkowsky,...
14What you're describing sounds a lot like grieving, I've been there both in my professional life and I've lost people that were close to me. My advice: focus on what you can do. Give yourself grace, and don't lose sight of your motivation. Make time to actually remember to do the things you enjoy that bring you peace and remember why it is worth it to live at all.
This is important work, but it's not going to help anyone to burn yourself out over it.
And guess what? Fact is that we're all already dead anyway. That's the curse of having been born at all, price of admission. You can't work your way out of it, and it's the same problem you're facing here on a different scale.
Take the time you need to grieve the life you thought you'd have. That's okay, and it's healthy. Then buckle up, look around, and remember that the world hasn't ended yet, so you should get on living like there's still something out there worth saving.
8And guess what? Fact is that we're all already dead anyway. That's the curse of having been born at all, price of admission. You can't work your way out of it, and it's the same problem you're facing here on a different scale.
Far future technology inevitably makes (involuntary) death a historical curiosity, and distributed backups make it technologically impossible for as long as the global civilization persists as a whole. AIs very likely move that future within most current lifetimes, and a cryopreservation interlude is available in some of the other cases.
Human extinction makes this moot, but it's not a certainty, and we can work to avert it. As such, death for the currently living (or the currently cryopreserved) is not a certainty, it's false that "we're all already dead anyway", that there's this magical "curse of having been born at all" that can't be defeated.
11[comment deleted]
8
the standard advice was given by CS lewis about living in the atomic age:
[...]
i have one more point, which i think is less often given, and i think fairly solid on its own grounds:
when doom is coming (climate doom, demographic doom, democracy doom, AI doom, etc), it may seem like the only legitimate response is hopelessness. hope is hard to find, dread and despair is easy. but these responses can only be had when there is some uncertainty about whether doom or success is coming. no one feels dread, nor hope, about whether the sun will rise tomorrow: your certainty is ~100%. you can't feel hopeful that smallpox will be eradicated; it has been, so you feel gratitude. it is actually only in uncertain situations where you can feel either hope, or dread, and you have to find it within you, and carry it, until it's over.
persist, despite the odds
6
I think it's possible that there are 15-20 more years of humanity in control (if prosaic LLM alignment work goes well), that it might be possible to hand off long-running projects to AGIs (where the initial human work shapes the purpose of the projects), and that permanent disempowerment is more likely than extinction (if prosaic alignment/control goes well and a takeover happens when AIs are either superintelligent or at high levels of industrial explosion). The 15-20 year figure can be cut short by the invention of superintelligence (architecture-rewriting strong RSI; as opposed to slow-learning prosaic RSI) at essentially any point, and I think it's likely that it will be, but then the caveat about permanent disempowerment being more likely still applies, in a takeover that's performed from a position of an overwhelming cognitive advantage rather than by approximately human-level AIs using an approximately human-scale industry (which is what happens if prosaic alignment/control fails before superintelligence or high levels of industrial explosion).
So still very likely doom, but it seems different from your picture of doom, and motivates long-running projects or prosaic alignment/control rather than primarily very short-term things that win dignity points. The motivation on a ban/pause being much better seems about the same. And it's possible that the prosaically aligned LLM/RL AGIs do something right at the superintelligence (architecture-rewriting strong RSI) phase transition, which is where the unfinished long-running projects might steer the outcome (of what happens within the crumbs that the future of humanity is left with), though I still don't see how that could realistically avert permanent disempowerment (in the sense that almost all resources go to the AIs rather than to the future of humanity). (Note that there are people like Musk who seem to describe/consider permanent disempowerment as "a good outcome, an age of abundance", which is confusing when
5
I’m pretty much the same. When that sense of impending doom creeps in, this is what I tell myself:
Remember, unmanageable emotions won’t help you solve anything—they are the problem itself.
Go jerk off, get some sleep. If you still feel off afterward, consider medication, socializing, getting some sunlight, watching your diet, exercising, staying hydrated, or just changing your environment.
Then, take stock of the evidence behind your beliefs and seek out information from a wider variety of sources.
"Reduce your regrets if the world were to end shortly, but not in ways that would make you regret to hear that the world wasn't ending."--Shortform by Eliezer Yudkowsky
5
Worth reading
3
I have a lot of compassion for this. It’s important to process your emotions; journalling, going on long walks, and talking to friends about it are good ways to do this.
Personall I feel like this CS Lewis essay helped me a lot through a similar time. https://livingbulwark.net/wp-content/bulwark/april2020p7.htm
2
This is all sensible. Life can be extremely destructive. People can be metaphorically or literally buried alive. They can be broken by cruel choices they face, or perish through the irresponsibility of others.
I would say there is no guarantee that incompletely aligned superintelligent AI (I call it that because the frontier companies are trying to align their creations and do have some level of success so far) will manifest as human extinction. I usually say that what's more guaranteed is takeover, because the "intentions" of superintelligence should be capable of dominating the world, the question is simply what those intentions would be.
But for now, even that is not guaranteed. Maybe the great powers will get together and make a worldwide ban. Maybe the wheels will fall off the data center economy and we'll be living in Mad Max with drones. Maybe aliens step in, maybe this is all a dream, maybe AI progress conveniently stops at this level... well, that one seems least likely of all.
2
You’re not currently at risk of standing on your fire escape on Judgement Day and watching swarms of AI killer drones descend on my city and thinking "fuck, I shouldn’t have done that". It’s a low bar, but if we can get everyone who came in with dreams of saving humanity from AI over that bar and into your high company, we’ll be a lot closer to the “see the suicide button and not press it” good ending.
I've found LLMs useful as an assistant for coding, writing, research, bizdev advice, and general ideation, but even the almighty Fable 5 is currently very unhelpful to me as an amateur game designer.
I might write a whole blog post about this. For some reason LLMs just don't seem to understand what makes a game mechanic "good"? When I present my existing ruleset to Fable 5 and ask it to create new content or design a new mechanic according to specifications, it never has good ideas.
In the set of disciplines in which I have some working knowledge (which is admittedly very few), game design has by far the most significant taste gap between skilled humans and the best LLMs.
But again, I'm just an amateur. I'm curious what professional game designers would say to this.
1"Saruman," said the wizard, "I come to you with grave tidings. The Enemy is moving. RSI approaches. The GPT-8 Agents are abroad."
"Is that all the news you have for me, then?" asked Saruman, hobbling away with a dismissive wave.
"Is that timeline not short enough? AGI is self-improving at last!"
"We can deal with AGI ourselves, Gandalf. You, and I. One way or another."
"What talk is this? What are you saying, Saruman?"
Saruman grinned. "It is time for us to choose, Gandalf. A new age is upon us. A new power is rising. Nothing that alignment or mech interp or MI...
1