@habryka and I recently spoke about his plans to improve the AI safety funding ecosystem with a better S-Process platform, and my new incubator for EA/AIS software projects, Surplus (since launched; apply now!)
We also cover: hot takes on different funders; what kinds of founders might succeed in the age of vibecoding; whether to do direct work or go meta; and what we respect and criticize in each other. Watch along here:
Oli: "Philanthropy is one of the most cursed games in existence... The default outcome of what happens when rich people try to do philanthropy is that they think about starting a foundation, they imagine hiring someone on the market and ask themselves: who am I going to show up and feel comfortable trusting most of my net worth to? That doesn't make any sense.
And so what they often end up doing is making a family office. The only way to solve this principal-agent problem is to choose someone where you have a very strong pre-existing relationship — someone you know isn't going to try to extract as much money as possible, or even just extract 5% of your money, which will turn out to be many millions of dollars. That trust bottleneck is just very intense."
Thoughts on specific funders: SFF, CG, Longview
Oli: "I'm thinking it would be good for there to be some kind of funding system that people can distribute money through that requires less trust, is less bottlenecked on building your own foundation, and has access to a substantial amount of deal flow.
I historically built the S-process and collaborated a lot with SFF in building that. I'm generally very proud of the work we did. But my current best guess is that SFF has been risk averse and opinionated about what kind of grants they allow, which kind of prevents it from really coming to fruition."
...
Oli: "GiveWell historically did something I thought was very good. Holden built an institution where he said: you know you can distribute money through us because we're going to be extremely transparent. We're going to evaluate things in a way where you can follow our reasoning and see our receipts...
And I think that's one of the things that enabled Dustin to trust them and put them in a position to distribute a much larger amount of money via Open Philanthropy and Coefficient Giving. But as far as I can tell, they've now largely abandoned that as the central project."
...
Oli: "Longview kind of never even tried. They're just trying to be an institution that somehow builds such a strong trust relationship with its funders that it can overcome the principal-agent problem easily, and I'm not that optimistic about it...
I also think that partially it's just kind of a con, in the sense that a lot of EAs are giving strong endorsements of Longview because they're worried that otherwise the money won't go to charity at all."
On professional grantmaking
Oli: "I don't really believe in professional grantmakers as a role that benefits from specialization — I think it's deeply harmed by the fact that you don't get to experience things.
Austin: So, this is a pretty big critique of the whole CG model.
Oli: That's right... the only thing that has historically kept CG on rails is that at least many of the people there actively published their ideas; they were part-time cause-prioritization researchers and part-time grantmakers. I think that was a balance that could work. But CG has completely stopped doing that.
Nobody at CG runs the kind of blog that Holden used to write or Joe Carlsmith used to write."
What Surplus is incubating
Austin: "The three main categories of things I'm interested in: one would be AI for epistemics and coordination — a broad class of things using LLM capabilities to help individuals, groups, or society as a whole reason better, think better, work together better.
Second would be public-facing content, stuff like AI2027, 80,000 Hours, or BlueDot — sites that take ideas people in our community have thought about and try to disseminate them more widely.
Third would be more internal-facing infrastructure — a better grants process would be an example, and historically things like LessWrong, Manifund and other internal coordination tools."
On who should and shouldn't go meta
Oli: "I'm tentatively excited about this project for you, which is actually a huge bar — because for years I've been on a crusade against incubators. I've advised maybe two dozen people to please not start an incubator.
The usual reason I say that is because they're trying to incubate in a domain where they've never produced any valuable work themselves."
...
Austin: "I often get these very young undergrad people who have recently seen something like Nan Ransohoff's article on third-wave philanthropy and thought, "Oh yeah, there's going to be a lot of money, I should do something about this." And I'm like, you might not be the person who is well-placed to do something about this...
These people should start orgs, or projects that can take funding if the projects do well. Because the scarce thing when there's a lot of money is orgs, projects, people, and good ideas. Doing that directly seems more good than doing meta-level stuff."
On for-profits and feedback loops
Austin: "Many kinds of for-profits encounter more feedback from reality because they're trying to get revenue and have to generate some value for the world, compared to many kinds of nonprofits that are research-oriented or vaguely community-building. I think it was a great thing when EAG started charging people for tickets, because it at least grounds some amount of value that EAG is providing.
A second reason to favor for-profits is the huge torrent of funding coming up. Some of these dollars will end up in nonprofits, but the nonprofits themselves are going to need a lot of tooling, infrastructure, and services that can be sold to them by for-profits. In that ecosystem, it will probably be better if orgs are charging each other what seems like fair value, as opposed to everyone just holding hands and giving everything away for free. Money helps solve coordination problems."
Chapters
[0:00] Critiques of SFF's grant process
[2:26] The SFF application process
[3:40] The speculation grant freeze for advocacy orgs
[5:04] A lower-trust, more transparent funding process
[6:53] How the S-process works
[10:54] Naming and communicating the value to funders
[15:51] EA philanthropy and the principal-agent problem: Open Philanthropy, Longview
[21:28] How much funding is coming
[22:33] Surplus: the incubator
[24:37] Why for-profits over nonprofits
[27:11] The ideal founder profile
[30:26] Whether writers can found startups in the vibe-coding era
[31:45] Monetizing public communications projects
[42:09] Oliver's case for the incubator
[44:04] On professional grantmakers
[47:36] Whether infrastructure work is more direct than safety research
[51:08] The case for a better AI safety journal
[53:17] Mutual feedback
[1:01:01] How to help: LessWrong, Surplus, and the S-process
Yeah, so I think this is going to be a pretty casual conversation. I want to start with the funding side, like better SFF.
Oliver Habryka
I gave you the rough summary the other day. Roughly, I'm thinking it would be good for there to be some kind of funding system that people can distribute money through that requires less trust, is less bottlenecked on building your own foundation, and has access to a substantial amount of deal flow.
I historically built the S-Process and collaborated a lot with SFF in building that. I'm generally very proud of the work we did. But my current best guess is that SFF has been risk averse and opinionated about what kind of grants they allow, which kind of prevents it from really coming to fruition.
When you say SFF has — that's the management team of SFF? That's right. It's a small board — Critch, Ethan, and Eric Rockstead.
They run the thing, but they have a lot of opinions. They recently had a policy where if you do anything that remotely looks like advocating for violence, or tell jokes in which a fictional character advocates for violence, you are banned forever from receiving SFF money. These kinds of rules just end up being used for a huge amount of chilling effects and a huge amount of annoying political prosecution.
It just makes me quite worried about the future of the fund.
Austin Chen
Maybe one way I would frame it is that when I first heard of SFF, I was like, oh, there's this beautiful thing where lots of different people with lots of different opinions can collaborate together, but also fund specific things that they like themselves. That's what the recommenders of the S-Process, and also the speculator grantors, are empowered to do.
But you're also saying that even though there's this beautiful schematic, in practice the people organizing the S-Process have a lot of say, and that shapes things.
Oliver Habryka
There are a lot of opinion constraints. The application is extremely difficult and confusing, and I've been trying to fix the application for years and I can't. It's a huge bottleneck on getting good people the ability to get money via the process — just procedural commitments that don't really make sense to me.
The SFF application process [2:26]
Austin Chen
So I applied for SFF in the most recent round for Manifund. Mox applied on behalf of Mox. And also a lot of people I knew came to me asking if Manifund could be the fiscal sponsor for them, and they did it. So I have a fair amount of experience with the application from the applicant side.
And yes, it is the most complicated application process I've ever seen.
Oliver Habryka
It's complicated for completely random reasons. You have to give answers in three forms, fill them into a Google Doc, and also fill them into the general application. And if you forget something, you can't get a grant. You also have to fill out the speculation grant application, which asks why you need money quickly — which doesn't make sense — but you have to fill it out because if you don't get a speculation grant, you're not eligible for the main round.
Austin Chen
Suffice to say the grantee experience has many aspects that could be improved upon, even just filling in the application. That said, there are some things SFF does quite well — it makes unrestricted grants, which I very much appreciate. Paul Graham has a great write-up on why to give unrestricted. And the speculation grants do come relatively quickly, which is nice.
The speculation grant freeze for advocacy orgs [3:40]
Oliver Habryka
But that was actually the big thing that pushed me over the edge on doing this — SFF just blocked all speculation grants to all advocacy organizations. That was a huge deal because these organizations were all relying on the speculation grants. The whole point of a speculation grant system is that the speculation grantors can ensure grantors are not reliant on the very slow turnaround time of the S-Process.
What happened this round was that there were about 25 organizations that were basically told they were going to get the money through the process as-is. And then suddenly it was like, oh no, sorry, this is not okay — we don't want this to be part of the speculation grant process.
Austin Chen
Just for context, can you share some numbers if you're allowed to — how much total money was given through the speculation grants, and how much people are expecting for the final round?
Oliver Habryka
My guess is that the number of grants that have been delayed is around 5 to 6 million total — maybe less, maybe only 3 million. But I know of a single grant that's 500K, and there are a few others. So somewhere between 3 and 10 million.
Austin Chen
And historically, the rounds themselves are around 20 million approximately?
Okay, so you saw this and thought maybe you want to do something better. And I imagine you also want to get other funders besides just Jan himself involved — having them participate, put in their funds, and help select recommenders, that kind of thing.
A lower-trust, more transparent funding process [5:04]
Oliver Habryka
That's right. What I really want is to make it so that if you're someone who wants to give away money — whether it's Dustin, or someone at the labs with a bunch of equity — there's a process they can join where they can observe what would happen if they were to give money to it. You can show up, see the applications, see the organizations, and ask: what would happen if I were to distribute $2 million through this process?
You get to see what the result would be — who you're deferring to, what they've said about relevant applications. Basically, at the last step, you get to decide how much money to distribute. That's a much lower-trust commitment than the default situation with a foundation, where you have to form an endowment, hire permanent staff, and build long-term relationships with organizations. I think that's a truly terrible experience if you want to make philanthropy easy.
Austin Chen
That's interesting — I actually wasn't even aware of how the SFF process works. And incidentally, I think I'm maybe one of the top 1,000 people in the world who cares about funding processes, and I still don't quite understand how it works.
Oliver Habryka
Their marketing has been really not very clear. Critch has been doing a good job with a few explanations in YouTube form, and they now have a demo app you can play around with. But a huge issue with the SFF process is just making the marketing clear and understandable — being clear about the value propositions, not just the mechanisms. Someone could derive the fact that you're allowed to go last as a funder and that this gives you a kind of assurance. But just telling people: hey, if you join this process, you get to know where your money goes before you spend it — in a way that's very difficult to achieve by almost any other means — that's the key message.
How the S-process works [6:53]
Austin Chen
Can you explain just a bit more about how that works? I had imagined the SFF process as: you pick some recommenders, give them the money, and the recommenders decide how to spend it.
Oliver Habryka
The key thing is that you work from the bottom up. You start with the applications. Then you have recommenders, and each recommender specifies a funding ordering over the organizations — essentially saying: here's the order in which I would fund these organizations. This is the one I'd give my first $10,000 to, then this one, then probably this third one. So they all specify a funding ordering.
The whole point is that they specify this ordering over a relatively wide range of money, because they might get different amounts flowing through them. And if other recommenders want to fund the same organizations, they want to ensure that if things get bumped further down their ordering, they can still reach that point.
Then the funders come in and decide in what order they want to distribute their money to the recommenders. At the end there's a final slider where the funder can say: do I want to put one million dollars into my funding recommendation over the recommenders, or two million, or three million, or zero? You can just see what grants get made if you give a marginal million dollars.
And because everyone has specified funding orderings, this allows you to just simulate what happens. This is where the name S-process comes from — it's a simulation process. You just simulate what would happen if you were to distribute an additional million dollars, and you get to see the counterfactual right there.
Austin Chen
Is the funder who's deciding whether to commit the next million dollars trying to figure out what allocation between the recommenders to give those million dollars to?
Oliver Habryka
They have some tools. Basically, you can decide which recommenders you want to give your next million dollars to, and then you can see what the recommenders would spend those million dollars on. You can choose how much and which different recommenders you're happy with if you distribute money through them. And I think it's very valuable to the system that the funder moves last.
The funder gets to look in and see who has done good reasoning. You want to really avoid certain kinds of gaming where people avoid ranking an organization highly because they expect other recommenders to recommend them first. So you want the funder to take some responsibility for checking whether anyone is gaming the system. It's really useful that the funder goes last — but this also means it's easy for the funder to pull out at the end and say, "I thought I was going to give $2 million, but seeing the quality of the recommenders' reasoning, I'm either pulling out completely or only giving $200,000."
Austin Chen
Maybe in my head the ideal thing I would try to design from first principles would be something more impact cert-like, or more standard VC equity-like, where the LPs are funding funds that then fund specific projects. And then every once in a while, the LPs — the Dustins and the Anthropic donors — decide, based on the performance of these funds or the specific set of charities they're funding, which ones they want to give money to.
In some ways, it seems pretty similar to the S-process as described.
Oliver Habryka
Yeah, maybe. Basically, you have a recommender, and I'm not at all attached to a recommender being a person as opposed to a team. Currently, recommenders are all individual people, but there are some weird edge cases. For instance, there exists an AI recommender that's been in every round, and there's a mean recommender, which just averages the implicit utility functions of everyone in the process — and that's an interesting thing you could decide to defer to that isn't necessarily a person.
Naming and communicating the value to funders [10:54]
Austin Chen
Where do you think were your biggest constraints on getting this built?
Oliver Habryka
The biggest thing right now is actually just trying to communicate what it is. What on earth does it mean to participate in the S-process? Generally, my biggest bottleneck right now is finding a good framing and name for it.
Is it a platform? Is it a market?
Austin Chen
I suggested "Exchange" earlier.
Oliver Habryka
I think "Exchange" really leans into the impact certificate type thing, which has some good attributes, but it's trying to cross a lot of inferential steps in one big pile. So trying to figure out a good communication that is centrally focused on helping funders — donors and philanthropists — understand what the product is.
I'm very happy about the Lightspeed Grants Round, which we started a while ago, but it's very oriented around communicating that it's going to be a good experience for the grantee. That's not my priority right now — I really want to get to a point where I can communicate the benefits of it to the funders.
Austin Chen
Yeah, and I have some sense that if you do the classic startup-y things — start with a small round, run more rounds down the line that kind of show the work — then people will be like, "Oh, cool, I can commit a small amount of money, or just commit virtual money to this to see what would happen, and participate without even having to commit anything except time."
Time, to be clear, is a big commitment. But maybe you could get people understanding this better.
Do you have a sense of when tentatively you might be able to do the first round?
Oliver Habryka
Right now I'm just writing up a Google Doc. I'm hoping to send it over to funders who might be interested in participating. I'm hoping to do something in August — that would be ideal. If we open applications a bit earlier, mid-July would be ideal for a public launch.
We could maybe even get a bunch of the applications that have gone to the Long-Term Future Fund and the SFF so we can start even earlier than that. Then having the first grant round dispersed in August would be ideal.
Austin Chen
And do you have a sense of how many dollars would be the sweet spot?
Oliver Habryka
For the first round, something like $10 million would be pretty good. It's a substantial amount of money that solidifies this as a real institution, but isn't going all in in some super intense way.
Austin Chen
And then probably from there, regular rounds — do you think like every half year?
Oliver Habryka
I think quarterly. There's a balance I'm currently uncertain about: what do you want the balance to be between the round evaluation mechanisms and speculation grantors? Right now the SFF runs once a year, but that's less bad than it sounds because the speculation grantors get to distribute a decent amount of money outside of the round, and then they get evaluated retroactively.
Austin Chen
That wasn't even obvious to me. It's not obvious that somebody applying right now would have a good chance of getting, say, $50,000 if they had a good project.
Oliver Habryka
Yeah, you'd just get it immediately via speculation grantors. So there's kind of a buffer there.
But I still think the annual process is really very slow and very heavy, and as a funder it's a pretty terrible experience because you only get one round a year to decide how to distribute the money. Currently I want to do quarterly, and then as things get crazier as a result of AI taking off and various other things, I'd expect to shorten that — eventually probably more like a monthly settlement, with a rolling application process.
As a funder, make it easy to copy over your preferences over recommenders from last round and adjust them a bit, as opposed to starting from scratch every time.
Austin Chen
One of my favorite sayings is: if it hurts, do it more. It's an agile kind of saying, usually around merging code. But if the funding process hurts and is painful, if you do it more, you'll be forced to do it better. More feedback.
Yeah, my current sense is that being a recommender is also quite painful because you have to read hundreds of applications.
Oliver Habryka
The most insane thing I absolutely don't understand is that the SFF really wants every recommender to read every application. And I'm just like, that is insane. Why would you do that? Most of these are random applications that nobody should have to read.
Recommenders should be able to specialize in certain domains. You could say, look, I don't know how to evaluate this weird biodefense stuff — I'm not going to look at those. I want to focus on a certain domain where I have expertise. It just doesn't make any sense that all of them need to evaluate all of them.
This is one of those procedural commitments that just makes running the S-process much, much harder.
Austin Chen
Yeah, I totally agree. That doesn't make any sense to me either.
Did you want to say anything about other funders, particularly Open Philanthropy, Longview, or Manifund?
EA philanthropy and the principal-agent problem: Open Philanthropy, Longview [15:51]
Oliver Habryka
I think philanthropy is one of the most cursed games in existence. You have people who, via entrepreneurship or sometimes investing, amass a huge amount of money that they've paid a lot of attention to — how they grew it, what they did with it. And then the usual thing is they're like, now I want to do something good with that money.
They don't have much ability — they don't want to pay that much attention to exactly what is happening with their money. They're usually busy with other projects. And now they're trying to hire someone to hand all of their wealth to, to distribute it to a set of projects that are hard to evaluate and where the metrics are much less legible than market feedback.
The principal-agent problem is just extremely difficult. The default outcome of what happens when rich people try to do philanthropy is that they think about starting a foundation. They imagine hiring someone on the market and ask themselves: who am I going to show up and feel comfortable trusting most of my net worth to? That doesn't make any sense.
So what they often end up doing is making a family office. The only way to solve this principal-agent problem is to choose someone where you have a very strong pre-existing relationship — someone you know isn't going to try to extract as much money as possible, or even just extract 5% of your money, which will turn out to be many millions of dollars. That trust bottleneck is just very intense.
I think GiveWell historically did something I thought was very good. Holden kind of built an institution where he said: you know you can distribute money through us because we're going to be extremely transparent. We're going to evaluate things in a way where you can follow our reasoning and see our receipts. We have this very intense commitment to giving money on the basis of cost-effectiveness — figuring out the most cost-effective grants that can help the world the most.
Over many years, they built a reputation of being an entity you can trust to distribute money on the basis of cost-effectiveness. And I think that's one of the things that enabled Dustin to trust them and put them in a position to distribute a much larger amount of money via Open Philanthropy and co-funding giving.
But as far as I can tell, they've now largely abandoned that as the central project. Now when you talk to them, they say a huge amount of what they're going to give away is stuff that puts them in a better position to direct future funds — stuff oriented around protecting and maintaining the brand and reputation of their funders.
Those things put you into much more of this principal-agent problem category — you no longer have a clear commitment to evaluating cost-effectiveness. You very rarely have any kind of inspectable public write-ups that people can cross-check. I think they're just going to be thrown into the midst of a lot of the classical principal-agent problems, and the forces of that are pretty messy.
Longview kind of never even tried. They're just trying to be an institution that somehow builds such a strong trust relationship with its funders that it can overcome the principal-agent problem easily, and I'm not that optimistic about it. Historically they've been extremely limited in what grants they can make because they really need to select from a subset of grants that look credible and legible in a way that just often gets rid of 99% of the impact.
There have been very few Longview grants I've been excited about historically.
Austin Chen
I mostly agree. I'm very confused why it seems like the Anthropic default — or among the set of Anthropic employees with liquidity in the current situation — is that they're planning on giving it mostly to Longview. I don't know.
Oliver Habryka
I mean, what's the alternative? That's the big question. Like, what else are they going to do?
But I also think that partially it's just kind of a con, in the sense that a lot of EAs are giving strong endorsements of Longview because they're worried that otherwise the money won't go to charity at all. So they're willing to say things like, "these are amazing advisors, these are really trustworthy people" — when I think the right answer is to say, hey, this is a really, really cursed problem.
There's a decent chance these people will be in a really, really difficult position where they'll be very tempted to recommend grants that sound good to you when you think about it only a tiny bit.
They can get more money from you, use that to establish more of a reputation, and there's this whole parallel game. These people are probably going to give you better advice than others who are also in this cursed problem class, but you should view them with a lot of suspicion — this is one of the most difficult principal-agent problems that exists. But that's just not what people are saying. People are saying these people are extremely trustworthy and great and you should give them your money. I think there's a thing going on where people want to make that recommendation because the money is very dead otherwise — it's not going to go to charity at all.
I'm currently expecting that a lot of Anthropic employees will be pretty burned in a year or two. When they experience these dynamics, it becomes very clear that a huge amount of grants are never presented to them because people are worried they'll find them weird. People will see that there's a lot of politics behind the scenes about which grants get put in front of them. People will see that there's a lot of weird reputational management — like, 'I can't make this grant because another funder would be angry at you if you made this grant.' That's really not what I want.
How much funding is coming [21:28]
Austin Chen
In some sense, I myself am not that worried about the way money plays out this year, and more worried about how funding plays out in the next year and the year after that.
Oliver Habryka
This year, I think everyone will struggle to just give money to anything, and a lot of things will be spinning out. I think we're going to see a huge amount of funding sitting in bank accounts — a lot committed, but very little actually distributed. Then in the following year and after that, I'm pretty worried about how that will play out, especially with an IPO perhaps.
Austin Chen
I heard some crazy numbers. Someone was saying maybe something like a hundred million dollars this year, then a hundred X that the next year, and a hundred X the year after that.
Oliver Habryka
The second 100X thing sounds tricky, but the first 100X thing sounds genuinely plausible to me. Yeah, 10 billion — that seems very plausible. 1 trillion, less plausible.
Austin Chen
Yeah, and then there are questions about how Anthropic itself, OpenAI, and other things play out. Maybe we can switch over and talk a little bit about incubating projects.
Surplus: the incubator [22:33]
Oliver Habryka
You'd like to see more interesting infrastructure and software that people with an altruistic mission in mind try to build to make the world a better place, and you've been thinking about running a fellowship of some kind.
Austin Chen
I'd maybe call it an incubator rather than a fellowship at this point — to be a little closer to Y Combinator than to MATS, perhaps, in terms of seniority or the kinds of outcomes we're hoping for.
The three main categories of things I'm interested in: one would be AI for epistemics and coordination — a broad class of things using LLM capabilities to help individuals, groups, or society as a whole reason better, think better, work together better. Second would be public-facing content, stuff like Asterisk, 80,000 Hours, or Blue Dot — sites that take ideas people in our community have thought about and try to disseminate them more widely. Third would be more internal-facing infrastructure — a better grants process would be an example, and historically things like Manifold and other internal coordination tools.
So those are three categories of projects I think I could help out a lot with. I'm open to other software projects and even non-software projects — I've done things like Manifest, which aren't very software-shaped.
Oliver Habryka
We made our conference largely as an app — though I think that's mostly not true. The people and the branding matters a lot. But I do think frequently, when I encountered a problem I'd seen at other conferences, I'd ask myself how I could solve it with software rather than just grit.
Austin Chen
That's a very good instinct to lean into, and especially easier now that vibe coding is so good. Some of my thesis about why to do this incubator now is that vibe coding is so good.
So many kinds of people who previously experienced problems can now just go hack and solve them themselves. One pointer to this is maybe Aella with Glosso.inc. In years past, I think Aella’s worked on other kinds of products that had some of the same ideas, but being able to execute on it herself makes it much better.
Why for-profits over nonprofits [24:37]
Oliver Habryka
One of the things that is most interesting about what you're trying to do is you're trying to do this project, but you're also trying to take equity in the organizations, or at least as much as that is possible. Have you figured out how to do that? Many of these people will maybe want to do it as a charity — are you going to encourage people to incorporate as an LLC, explicitly not as a charity? How are you thinking about structuring that right now?
Austin Chen
Yeah, and maybe explicitly as a C-corporation, not an LLC — whether a standard C-corporation or a PBC — just because those have better standard investor terms. I think some of it is just going to be stated upfront: we're looking for for-profits, for these reasons.
The reasons, very briefly, being that many kinds of for-profits encounter more feedback from reality because they're trying to get revenue and have to generate some value for the world, compared to many kinds of nonprofits that are research-oriented or vaguely community-building. I think it was a great thing when EAG started charging people for tickets, because it at least grounds some amount of value that EAG is providing.
A second reason to favor for-profits is the huge torrent of funding coming up. Some of these dollars will end up in nonprofits, but the nonprofits themselves are going to need a lot of tooling, infrastructure, and services that can be sold to them by for-profits. In that ecosystem, it will probably be better if orgs are charging each other what seems like fair value, as opposed to everyone just holding hands and giving everything away for free. Money helps solve coordination problems.
Something donors don't often appreciate about nonprofits — or 501(c)(3)s — is that 501(c)(3) dollars can go towards for-profits in order to buy services. And nonprofits can invest in for-profits. That's something Manifold actually does a lot — we often help donors put money in as a seed investment into a new for-profit.
Oliver Habryka
So are you planning for the entity that runs this to be a nonprofit or for-profit?
Austin Chen
Tentatively, the entity that runs this will be Manifund, so it will be a nonprofit, which means we can hopefully fundraise from 501(c)(3) donors. The standard thing Manifund does when putting investments into for-profits is that we hold money on behalf of the donor, like a standard donor-advised fund. And then if there's an exit scenario, the donor gets more money to invest into new charitable entities or other donations.
Oliver Habryka
That's kind of reasonable.
The ideal founder profile [27:11]
There are two things I'm most uncertain about for the project. One of them is: what is the kind of person you actually most want? It feels to me like there are three big cultural anchors that produce very different kinds of projects. You have the very impact-motivated people who are there because they want to do as much altruistic good as possible. You have the YC-type people you want to attract because they're mostly there for entrepreneurship. And then there are people who are just thinking, I can provide infrastructure to a bunch of the Anthropic people who will be extremely rich — I don't care about any of this impact stuff, but if I can get rich helping people, I'll gladly do it. And then you also have the rationalists who are thinking a lot about AI and epistemics, about how we can use these systems to reason better. Whenever I'm imagining this fellowship, I don't know where you're going to end up — this feels like one of the core tensions, and it will also determine how much synergy there is between the people and whether they'll actually successfully cooperate and gain value from each other.
Austin Chen
Yeah. I don't really have a good answer to that.
I kind of think there's a lot of things I respect about all three communities. I would try to market the incubator as much as I can to people in all these communities, and probably accept people who do well to be kind of like the intersection of some combination of these three communities. I'm hopeful that it's a large enough intersection that I can find at least 10 people who feel excited by startup-y things and rationalist-y things and impact EA-y things.
Different people will be more strongly tied into some of these communities. My current expectation is that I'll have a harder time finding good people from startup land to participate in this, maybe because YC offers more money than I have to offer. And YC is much more well-known — they're kind of like the Harvard.
I'm hoping I can attract some of them by being like, this is more like OG YC — intimate, small, a 10-person cohort with a lot of really good people. Currently, as a digression, YC does not get the best founders because the 200 or 500 person batches — however big YC is nowadays — feel more commoditized. Yeah, that's a good word for it. Like, a process.
So I'm really hoping that some of the best founders of the generation, who are even just not that AI-safety-pilled or whatever, look at this thing and find it kind of interesting.
Oliver Habryka
And also, in as much as you can pitch it as having a much higher density of interesting people — because with YC, you're asking how interesting is the average founder you're going to find — you can say, no, look, we're going for a much smaller, curated group. How many people are you currently imagining to participate in this cohort?
Austin Chen
Yeah, probably between 10 and 20 founders. So that would be more like 5 to 10 organizations, depending on the breakdown of solo founders versus co-founders. But that's my sense of what the original YC size was as well.
A lot of my constraints — or not necessarily constraints, but what I'm trying to do — is copy the homework of OG YC.
Whether writers can found startups in the vibe-coding era [30:26]
One question I was thinking might be interesting is: would Gwern, for example, be a good founder for this thing? He recently wrote this thing about guardian angels — it's kind of like digital twins, the idea of creating an LLM-shaped system that tries to be you as much as possible, as opposed to an assistant that is helpful for you. I think it's an interesting concept. I would be personally very excited if Gwern wanted to work on his idea in my incubator.
But yeah, someone like Gwern — that kind of writer-person — is that now a good fit for founding, because vibe coding is easy, et cetera?
Oliver Habryka
Oh, that's really interesting. There's been a joke going on ever since LLMs were even remotely able to code, which is: is the idea guy going to take over Silicon Valley? Like, maybe ideas are the only thing that matters. I don't currently believe it — you really need the iteration, you need to be able to talk to users, and there will still be tons and tons of grind work where a huge advantage will be the willingness to get through it. But we'll see how it goes.
Austin Chen
The other thing I would currently try to index for when looking for founders is people like me, people like you — people who have had the experience of building, but also are embedded in these communities.
Monetizing public communications projects [31:45]
Oliver Habryka
One of the things I least understand is the second category you mentioned about public communications — that one feels like the odd one out when I think about it. Like, how are you going to make money? I ran AI 2027 and we made zero dollars from it. Maybe we've been thinking about selling the movie rights — you could probably squeeze $100,000 out of those. It's a surprisingly popular media entity. But also, do you even want to?
Austin Chen
Yeah, I agree, it seems kind of hard. I mean, there's some ideal world where we have a new, great funder system that's happy to retroactively pay back people for doing good projects.
Oliver Habryka
In a world where you could potentially get a bunch of money from writing — which is proportional to the influence you get and the importance it plays in the world — that could be a huge shift and would make it more possible. But to ask people to work on a startup building that is kind of like building a bridge to nowhere, maybe, or maybe somewhere with a lot of gold, but it is very speculative.
Austin Chen
I do agree that the public comms category is the least monetizable in some sense, or at least the hardest to figure out how to get to revenue. There are also things about wanting to disseminate information freely. But there are some business models that do work — if you look at specific writers like Byrne Hobart, for example, he puts out a lot of free information and some amount of paid information.
Oliver Habryka
A Substack as a startup isn't crazy. Some Substacks are — we've never seen a billion-dollar Substack, but we are now in the range of seeing $10–20 million Substacks.
Austin Chen
I'm not quite sure yet about just being a Substack writer. Maybe we should just have really good Substack writers in the cohort. I don't really know — that's actually possible to me.
But also, if you look at things like 80K, I think there's room for something like 80K or Blue Dot to make money as a recruiting service. One of the major holes in the ecosystem — where there's lots of money flowing around — is finding good talent to hire when your org has a bunch of money but nobody to hire. I would encourage orgs like Blue Dot or Mats or 80K to think about whether, if you already have a pipeline and you're helping place people, you can ask the orgs to pay you every time you place a person.
Oliver Habryka
It's very tricky, because the whole value proposition of 80K is to tell you what the most impactful thing to do is — and suddenly you have this huge conflict of interest where maybe the most impactful thing doesn't pay you the most.
Austin Chen
Yeah, you don't want the failure mode of always placing people at Frontier Labs because those positions pay the most. I agree. But I remain open to it. You've got to capture some fraction of the value you provide somehow.
Oliver Habryka
That will always involve some amount of compromise and suboptimal incentives, because capturable value very rarely equals the full value you provide. But you still need to do it, otherwise you don't get to exist.
Austin Chen
Maybe a good thing to do would be to write down different funding models — how does Manifest make money, how does Lighthaven make money? I think those are two examples of models that actually have non-obvious business models but kind of work out, and it'd be worth exploring what goes into making those work. That's my hope for the second category, but fundamentally I agree — it's going to be hard to make a lot of money on those.
Oliver Habryka
You're currently imagining about two months for the program? Probably three months, starting late July.
Austin Chen
I was about to launch today, but then the Claude or Mythos team dropped something and I thought it was a bad day to launch — maybe I'll try tomorrow. Bad day to launch — I've got to redo my launch website with Mythos now. It's like the cheapest alpha just sitting on the ground right now.
Oliver Habryka
I'm curious if you had any thoughts on things you would personally look for if you were joining an incubator. I gather this isn't how you would do it yourself.
The thing I've always wanted out of YC, and that seemed most interesting to me, is that it's very clear my organization routinely faces extremely important decisions. I have friends and other people to talk to, but it seems to me that many of these problems are ones that many other people have faced before — core decisions like wanting to scale up, wanting to hire a lot, and how to do that without destroying your organization.
I would genuinely be quite interested in talking to a bunch of people who have been through this, who have dealt with VC stuff, to help me understand how they handle negotiations with funders and things like that. There are a lot of norms — or rather, just practical matters — around how these negotiations work that seem very helpful to understand.
High-stakes business advice has felt like a thing that's just very hard to get otherwise. Maybe I can post more frequently on Twitter, but often this is stuff that's just very hard to talk about publicly and hard to navigate. That's one of the things that has always attracted me most to YC — just having access to people who've seen a bunch of stuff in business, so I can go to them and say,
"Hey, I have this huge conflict between two of my employees that's threatening to tear apart the organization." That's never happened to me, but I've seen it happen to other organizations. Getting advice from people who have navigated those kinds of really tricky situations is incredibly valuable.
Austin Chen
That's really interesting. I kind of see you as a big fish in a small pond, where lots of people currently go to you for advice. I remember recently somebody on the board of Manifold asked you about a related question. I hadn't thought about it, but it totally makes sense that you'd want to get advice from people who have seen even bigger scales of problems.
Oliver Habryka
As a CEO, it's pretty common for CEOs to have other CEO friends whose relationships are comparably strong to those within their own organization, just because CEOs face similar problems. It's very hard for someone to understand the kinds of issues a CEO faces if they've never faced them themselves. Having more access to a community of CEOs seems very valuable to me.
The vast majority of people I get to talk with are just researchers. I know maybe five CEOs — it's not that many.
Austin Chen
I kind of wonder if there are enough sizeable organizations in the AI safety space that you could pull together, like Beth Barnes and the Apollo people or something, and get answers to some of those questions. Though I feel like the research orgs aren't quite right for what you have in mind.
Oliver Habryka
That's right. I frequently have a sense that something doesn't feel quite right. I do believe that Silicon Valley has successfully developed a real lineage of management knowledge. If I went through YC, there are people I could talk to who have been partners and have been thinking about these problems for two to three decades.
A lot of people have sampled from 100 to 200 perspectives over that time. I'm very confident that almost everyone around me is just making it up. Very few people are really tapping into a longer lineage. This is obvious to me because I'm a huge fan of the Toyota Method and Lean Startup type thinking.
There's not a single other CEO around us who feels to me like they really get it. I don't think Buck gets it, I don't feel like Beth gets Lean Startup stuff at all. I just look around and think maybe people have different management philosophies, but I would really like to tap into a lineage more exposed to that kind of thinking, because I think it's the most successful lineage in the history of humanity for creating successful organizations.
Austin Chen
One person that comes to mind is Steve Newman. I don't know if you've met him.
Oliver Habryka
Oh, that makes so much sense. Somehow Steve Newman feels so chill.
Austin Chen
He is — he's just good. And it's top of mind because recently he was working with my wife Rachel on the Golden Gate Institute. Steve and also Taren, who's the CEO of that org, are both very excited about AI safety related things and have a lot of experience.
Oliver Habryka
Yeah, that sounds great — I should talk to them more. And I think more people are going to be like that, having worked at high-powered organizations and now wanting to do something about AI. I'm pretty excited about pulling together a network like that.
It's going to be hard to compete with people who have been YC partners for 10 years, but it's not implausible that when I look at the first cohort, everyone who's building something there could actually be a good peer in navigating these situations.
I hope that you get many first-time founders, but also getting second- to third-time founders — I think that's a real thing.
Austin Chen
I don't quite know currently whether I should be angling for more first-time or more second-time founders.
I implicitly feel like the terms I can offer are going to be embarrassing — quite tricky. A second-time founder who sold a $100 million organization might ask, do I really want to take $100K for 10% of my company?
Oliver Habryka
I wonder whether that's a strong argument for doing more variable terms. It has of course helped YC enormously to have standardized terms, but it's plausible that you're in a more competitive market. Because the early singularity startup founder market will include a lot more high-powered people, there will be many in strong negotiation positions. So for some of them, you might just want to offer much better terms — though it seems pretty tricky to navigate socially and puts more adversarial pressure on you.
Austin Chen
I really hate negotiating, but I'll think about it. I was hoping for just one standard term. But YC does, for very prestigious or third-time founders, allow for different terms — they just don't advertise it. Of course they don't want the word getting out.
Nobody at YC has told me this, so this is speculation.
One topic I'm interested to hear your thoughts on is: why work on funding something versus incubating something versus directly doing the thing yourself?
Oliver's case for the incubator [42:09]
Oliver Habryka
I'm tentatively excited about this project for you, which is actually a huge bar — because for years I've been on a crusade against incubators. I've advised maybe two dozen people to please not start an incubator.
The usual reason I say that is because they're trying to incubate in a domain where they've never produced any valuable work themselves. I think of things like the FLI fellowship — they were trying to get people to do interesting AI-for-epistemics work, but had never done any themselves. Similarly, I've seen research organizations want to build an incubator for good research when they'd never produced any interesting research themselves. I just don't think that's a good idea.
This was in some sense the central secret sauce of YC — it was an incubator by founders who had experience founding things. You should only ever consider starting an incubator after you've actually started one instance of the kind of thing you want to incubate. So you've got to start with building something yourself.
I feel slightly better about being a funder rather than an incubator — though it still seems very risky. Can you fund work you've never done yourself? It's tricky, but slightly less doomed, because you have a much less intense effect on the culture of an organization and you're not trying to tell people how to run things. You can still become a huge bottleneck, but it's slightly more reasonable to figure out how to make a binary funding decision using deference chains and asking others for advice.
On professional grantmakers [44:04]
I still wouldn't become a funder if I'd never done good work in the space. The only reason I feel comfortable moving into funding is that I've always advocated for funders who also have a double identity. One of the things I'm most excited about for both the existing process and anything I want to build is having many recommenders who actually work in the space and are actively connected to it.
I don't really believe in professional grantmakers as a role that benefits from specialization — I think it's deeply harmed by the fact that you don't get to experience things. So this is a pretty big critique of the whole CG model.
Yeah, I don't think professional grantmaker is the right role. The only thing that has historically kept CG on rails is that many of the people there actively published their ideas — they were part-time researchers and part-time grantmakers. I think that was a balance that could work. But CG has completely stopped doing that.
Nobody at CG runs the kind of blog that Holden Karnofsky used to write or Joe Carlsmith used to write and stuff like that.
Austin Chen
Yeah, I mostly agree — this is why I still like re-granting a lot. The concept of re-granting is that you give money to people who are very part-time grantmakers and mostly doing their research, field building, or whatever they're really good at. And then if they find good opportunities, they can apply money towards that, while still being grounded in the actual work.
Oliver Habryka
So I guess, answering your question about how to trade off — just go and build something that actually provides value that people would be willing to pay for. Even in a nonprofit space, try to do something where you find a customer, whether that means you can clearly demonstrate impact, sell a concrete product, or charge admission rates or anything like that.
After that, I would consider doing something in the incubation or funder space. The multipliers are enormous — it's obviously the case that YC has had a much bigger impact. It's very hard for me to imagine Paul Graham doing something that would have had as much effect on the world as starting YC. It has shaped Silicon Valley beyond what I think any startup could have realistically done.
So obviously, often it's the right choice to work in infrastructure and things like that.
Austin Chen
Yeah, but I often get these — I don't know if you've been getting them — very young undergrad people who have recently seen something like Nan Ranselhoff's article on third-wave philanthropy and thought, "Oh yeah, there's going to be a lot of money, I should do something about this." And I'm like, you might not be the person who is well-placed to do something about this.
Oliver Habryka
In some sense, I think they're well-placed to do things that other people might want to pay for. But they shouldn't try to start a private fund — that doesn't really seem like the right response to this situation.
Austin Chen
Yeah, I mostly think — and I'm kind of talking my book here — that these people should start orgs or projects that can take funding if the projects do well. Because the scarce thing when there's a lot of money is orgs, projects, people, and good ideas. Doing that directly seems like more good than doing meta-level stuff.
Whether infrastructure work is more direct than safety research [47:36]
And that consideration applies even for you and me — we can do infrastructure-level meta stuff, or we can just directly do a specific thing that seems really good. I don't know if you try to think about how to navigate that personally.
Oliver Habryka
Especially for existential risk — there's a straw man I think people have that I really don't get, where they talk about "direct work on x-risk." I'm like, what do you mean? I don't know what direct work on AI safety is. Do you mean research?
I think the work I do has a much shorter causal chain to having good AI outcomes than people who are trying to solve mathematical problems adjacent to AI safety — because they then need to tell a story. The story for the vast majority of AI safety researchers routes through the existence of platforms like LessWrong. How can LessWrong have a longer causal chain than the research itself? The research would have nowhere to go without an active field.
The research would have nowhere to go without these things. These are just causal chains that I think are ultimately actually shorter than the vast majority of what people conceptualize as direct work. AI safety and AI existential risk is the kind of cursed problem where it's very hard to have anything that feels like directly solving the problem.
Sometimes people imagine that if they're at a lab, their impact can come from directly solving the problem. But that's really not the case. The vast majority of people at a lab will find that their impact comes from the degree to which they're creating a culture that is less likely to deceive themselves about the difficulty of the alignment problem, which will enable advocacy for policy changes at a crucial moment in time, which requires other people to be listening at the right time, to then be proposing the bill that can get enacted so we can get international cooperation to solve it.
And you're like, yes, that's the same long causal chain as everyone else. I think people are so attracted to this idea that they rationalize a huge amount of work as being useful when I think it's basically useless.
There's a huge amount of very local alignment research where people say, "I made the alignment benchmark go up, that must mean I helped the problem, I solved the problem." I really don't think the action is in the alignment benchmarks. The alignment benchmarks aren't tracking the problem well enough.
That's really not where I think almost any of the likely failures of this whole situation will come from. I agree it can feel concrete and specific, but as much as you're hoping to have your eyes on the ball, I don't think there's a sense in which it's more direct to work on a problem.
Austin Chen
I had a very toy model of, oh yeah, the technical AI safety researchers, that's direct work. And then the field builders, like you and me, we're doing the meta work of helping these people. But yeah, I can see it.
Oliver Habryka
No, the safety researchers rely on institutions like you and me to exist to even do their work. So at the very least, we're at the same level of abstraction. And I would say that I'm placing myself much, much closer in terms of uncertainty.
When I'm doing things like improving the quality of all the AI safety research at once, I get to take distributed bets, I get to make changes that pay off much more quickly than the average safety researcher. When I set up changes about how content gets popular on LessWrong or the Alignment Forum, or various UI improvements, I get to see the differences in how many people read that stuff and where its impact is at a scale of weeks relatively quickly.
A researcher will release a report every six months to two years, and they need to rely on there being infrastructure to publish that kind of stuff.
The case for a better AI safety journal [51:08]
Austin Chen
Kind of on that note, one of the ideas floating in the back of my head is a better AI safety journal or something like that. I think the Alignment Forum is reasonable and LessWrong is very good, but neither of these things are quite the thing yet.
Oliver Habryka
I'm super excited about that. If I had more time, it would be at the top of my list of projects I would want to do.
LessWrong unambiguously has post-publication peer review in the sense that you publish your stuff and then everyone will tear it apart in the comments. It's much better at that than the vast majority of ML and AI academia peer review. The post-publication peer review is genuinely high quality and tracks quality to a really substantial degree.
But the key thing is that right now it relies on a relatively fragile social status game where you can get reputation by publishing highly upvoted comments, but the visibility of comments is kind of mixed. You are building reputation, but only among a relatively small crowd who can really evaluate whether your comments are good. Having something where we're paying people money to do reviews of good work would be a big improvement.
That might be a lot of money, because reviewing good work is actually really valuable. I would be hesitant to have people specialize as reviewers too much — going all the way to being a full-time reviewer would be an issue similar to my concerns about starting an incubator or funder. But getting to a point where you're a half-time reviewer and see it as a major part of your professional responsibility to review other people's work is definitely where I want to get, and where the current field is really falling quite flat.
Yeah, I'd love something in that space. I've thought of a bunch of cool mechanism designs and ways to do stuff like that. But at the end of the day, it needs a lot of taste and a decent amount of communication and branding. And I'm just fully booked out.
Austin Chen
The first one that comes to mind as someone who's kind of doing this thing is Zvi. A lot of what Zvi is doing is summarizing a bunch of things and giving his takes, but also giving other people's takes.
Oliver Habryka
I loved Rohin Shah's alignment newsletter — it was great at the time. He ran it for about three years before he went to DeepMind.
Mutual feedback [53:17]
Austin Chen
What's one thing you admire about me and one thing you would criticize about me? And I'll do the reverse.
Oliver Habryka
Building good infrastructure is very hard. I think both Manifund and Manifold are just two great pieces of infrastructure that I'm very happy exist.
You're one of the few people I talk to who gets things that feel very basic to me — like PR FAQs, getting feedback quickly, failing fast, doing painful things more frequently. A lot of ideas around putting yourself in touch with reality as much as possible at every step to find whatever gradient is there. You feel like one of the few people who get that.
While also doing a pretty good job at keeping the big picture in mind. Most people in Silicon Valley, when I talk to them, take that mindset and just apply it to starting a company. They don't really apply it to asking what the most important problems in the world are.
My biggest criticism of you is that I think there's something you're missing about being careful around people. You're very high on my list of, like... "God, I bet Austin is going to like this crazy sociopath.” I don't know exactly what it is, but I think it might be related to how you relate to the company stuff.
I've been trying really hard to build a community with norms that maintain a certain kind of higher-trust relationship. I think you are trying to do something principled — you've aggregated principles around radical transparency and various things like that.
But I have long modeled the world such that if you're trying to shape the trajectory of AI, a lot of your effort should go into figuring out whether you're helping or hurting. If you don't assign substantial probability that you're making things worse, you're probably very wrong. It's a domain where the considerations are tricky and the right ethical principles are crucially important.
I think you have a very deep distaste for a certain kind of scrupulosity — where people obsess about whether they're doing good or causing harm in the world. I'm sympathetic to that, but I think you're not seeing a version of it that I actually think is good.
It is eminently possible to think about your role in the world, realize that you have caused grave amounts of harm, and use that to learn better — and to have a culture of people who take that question really seriously. I expect that if you keep building lots and lots of infrastructure, there will end up being a lot of people who make the world worse, and you will fail to have a culture that prevents that.
I think there's an underlying thing here where it's very, very tricky to have an ecosystem that does credit accounting in a way that is sensitive to whether you made the world better or worse, if you already have a substantial number of people who would be threatened by that becoming a fact about the ecosystem.
If you're in a world where there's a bunch of people who have made the world a lot worse — because they started frontier companies and really didn't try very hard to be careful — those people will form a very powerful coalition to prevent anything that could meaningfully create accountability. And so I'm definitely worried that in starting an incubator like this, there's going to be a bunch of people who make the world a lot worse, and it's going to make it very hard to have open conversations about that, because those people will very rationally understand that their standing in the ecosystem is going to be threatened.
And that's going to be very tricky.
Austin Chen
Thank you. I very much appreciate that.
I think you have many traits which I admire. One that I'm going to call out, that I think people don't notice as much, is that I think of you as being very service-oriented, really trying to help people. You often get in the weeds of trying to debug problems or fix things personally.
I was asking Lauren the other day who has stayed in the most rooms at Light Haven, and she said it's probably you — you do this all the time, you kind of dog-food them. I think the dog-fooding thing is related, and part of it is that you care a lot about making things really good in the service of serving people. I think it's a beautiful attitude and I would like to see more people adopt it.
Oliver Habryka
I'm glad. Now hit me.
Austin Chen
I think the thing where you got into a fight with Scott Alexander this year — that seemed really dumb. Maybe you kind of agree.
Oliver Habryka
Here's the thing you need to understand. If you start endorsing destroying the world, something really bad has gone on. You need to maintain a culture in which people can talk about this stuff. And if somebody is like, "How dare you attack these people who are carrying my banner? Aren't they trying to do good? How dare you even consider that maybe they're making the world a lot worse?" — I'm just like, no.
I cannot have a culture around me that is trying to protect the dignity of powerful people just because they might be scared to contribute. If we want to be morally serious, we need to maintain a strict standard. But I agree — it's tricky. I don't know how to navigate this stuff. I'm very confident that I'll look back at most of what I've done over the last few years in this domain and think I was an idiot. The problem is I just don't know which direction it'll come from. There's a lot of thrashing around when I'm trying to navigate a situation.
I'm definitely not being consistent in a way where I feel like I could behave this way if I had gotten the right answer. But I think I'm getting feedback about which things work, what the consequences of various attitudes in the space are, and I expect I'll figure it out and have slightly less dramatic public fights. Though maybe not — public fights are a pretty good way of working things out.
Austin Chen
Some of my favorite comments or writing from you have been public fights, like the coefficient giving thing. It really helped clarify my understanding of the space.
At the same time, I think it's probably good for you, your reputation, and the projects you care about to not be known even more as someone who fights a lot.
How to help: LessWrong, Surplus, and the S-process [1:01:01]
Maybe let's end on a classic podcast question: how can people help you out with the products that you're excited about? Either the S-Process thing or something else.
Oliver Habryka
A big one is we're finally in a position of financial stability to hire. Historically we've been the slowest growing organization of anything in the space, but I think we might want to hire three people this year. Three people. And as much as anyone watching this could be you, possibly — yeah, that's the big one.
Always contribute to LessWrong, contribute to the intellectual commons. I think that is at the margin very, very highly undersupplied still.
For the S-Process work — if you're a funder who has been intrigued by anything I've been saying about making grant-making more trustless, making it possible to get access to a huge amount of deal flow, now is a great time not only to potentially participate in the process, but also to shape it. If you reach out to me now and say you might be interested in distributing some money, you might be one of those early dogfood users where I build a product around you because you're my one source of feedback about how to do something good.
Austin Chen
And I think that's often a quite amazing experience. I can attest to that from my own experience, both with Lighthaven for Manifest the first time around, and more recently Waypoint for Manifest again.
Oliver Habryka
We have a gradient to optimize on, and that gradient might be you.
Austin Chen
I'll answer my own question too. Some things we're looking for: first and foremost, just great founders. Either if you yourself are interested, or you know people, or you're willing to help promote this — we should be launching very soon, so let people know Surplus is a place you can go.
Oliver Habryka
You never actually said "Surplus" in this whole session! Yes, it is called Surplus — that's the name of the incubator. And surplus.dev was the URL you were considering, which will probably be the official URL.
Austin Chen
We are also raising funding for Surplus. Doing the math, it's about 100K per founder, so maybe one to two million dollars total. Not that much in the grand scheme of things. I don't have any of it raised yet — I'm going to empty my coffers to make this thing work if I have to, but I'd prefer not to.
Oliver Habryka
In as much as you're open to donor-advised-fund-style arrangements, someone could come in with a million dollars and have essentially a Manifund stake with equity they could potentially do something with. Right — it could be equity, it could be a loan, depending on what the funders are looking for.
Austin Chen
I'm hoping that with enough money flowing around the ecosystem, somebody will look at this and think, yeah, this is exciting — I'm willing to pitch in a few hundred thousand dollars or something like that.
The third thing is that some people listening might be the kinds of speakers I'd be excited to bring in. You, Oli, would be a great person to bring in at some point. But also others who've started similar projects and maybe don't want to commit a lot of their time, but would come in and give a talk — those are the people I'm looking for.
And then general hiring as well — we're looking to hire around three people pretty soon for the combination of Manifund, Mox, and Manifest. I think the people who would be good working at Lightcone might also be good here.
@habryka and I recently spoke about his plans to improve the AI safety funding ecosystem with a better S-Process platform, and my new incubator for EA/AIS software projects, Surplus (since launched; apply now!)
We also cover: hot takes on different funders; what kinds of founders might succeed in the age of vibecoding; whether to do direct work or go meta; and what we respect and criticize in each other. Watch along here:
I've transcribed the full conversation at https://peruse.sh/ep/austin-chen-and-oliver-habryka-on-funding-incubating-project. (Beware: the AI makes notable edits for readability, sometimes distorting what the speaker meant. If specific phrasing is cruxy, listen to the audio.)
Selected quotes
The cursed game of philanthropy
Thoughts on specific funders: SFF, CG, Longview
On professional grantmaking
What Surplus is incubating
On who should and shouldn't go meta
On for-profits and feedback loops
Chapters
Referenced links
Full transcript
Critiques of SFF's grant process [0:00]
Austin Chen
Yeah, so I think this is going to be a pretty casual conversation. I want to start with the funding side, like better SFF.
Oliver Habryka
I gave you the rough summary the other day. Roughly, I'm thinking it would be good for there to be some kind of funding system that people can distribute money through that requires less trust, is less bottlenecked on building your own foundation, and has access to a substantial amount of deal flow.
I historically built the S-Process and collaborated a lot with SFF in building that. I'm generally very proud of the work we did. But my current best guess is that SFF has been risk averse and opinionated about what kind of grants they allow, which kind of prevents it from really coming to fruition.
When you say SFF has — that's the management team of SFF? That's right. It's a small board — Critch, Ethan, and Eric Rockstead.
They run the thing, but they have a lot of opinions. They recently had a policy where if you do anything that remotely looks like advocating for violence, or tell jokes in which a fictional character advocates for violence, you are banned forever from receiving SFF money. These kinds of rules just end up being used for a huge amount of chilling effects and a huge amount of annoying political prosecution.
It just makes me quite worried about the future of the fund.
Austin Chen
Maybe one way I would frame it is that when I first heard of SFF, I was like, oh, there's this beautiful thing where lots of different people with lots of different opinions can collaborate together, but also fund specific things that they like themselves. That's what the recommenders of the S-Process, and also the speculator grantors, are empowered to do.
But you're also saying that even though there's this beautiful schematic, in practice the people organizing the S-Process have a lot of say, and that shapes things.
Oliver Habryka
There are a lot of opinion constraints. The application is extremely difficult and confusing, and I've been trying to fix the application for years and I can't. It's a huge bottleneck on getting good people the ability to get money via the process — just procedural commitments that don't really make sense to me.
The SFF application process [2:26]
Austin Chen
So I applied for SFF in the most recent round for Manifund. Mox applied on behalf of Mox. And also a lot of people I knew came to me asking if Manifund could be the fiscal sponsor for them, and they did it. So I have a fair amount of experience with the application from the applicant side.
And yes, it is the most complicated application process I've ever seen.
Oliver Habryka
It's complicated for completely random reasons. You have to give answers in three forms, fill them into a Google Doc, and also fill them into the general application. And if you forget something, you can't get a grant. You also have to fill out the speculation grant application, which asks why you need money quickly — which doesn't make sense — but you have to fill it out because if you don't get a speculation grant, you're not eligible for the main round.
Austin Chen
Suffice to say the grantee experience has many aspects that could be improved upon, even just filling in the application. That said, there are some things SFF does quite well — it makes unrestricted grants, which I very much appreciate. Paul Graham has a great write-up on why to give unrestricted. And the speculation grants do come relatively quickly, which is nice.
The speculation grant freeze for advocacy orgs [3:40]
Oliver Habryka
But that was actually the big thing that pushed me over the edge on doing this — SFF just blocked all speculation grants to all advocacy organizations. That was a huge deal because these organizations were all relying on the speculation grants. The whole point of a speculation grant system is that the speculation grantors can ensure grantors are not reliant on the very slow turnaround time of the S-Process.
What happened this round was that there were about 25 organizations that were basically told they were going to get the money through the process as-is. And then suddenly it was like, oh no, sorry, this is not okay — we don't want this to be part of the speculation grant process.
Austin Chen
Just for context, can you share some numbers if you're allowed to — how much total money was given through the speculation grants, and how much people are expecting for the final round?
Oliver Habryka
My guess is that the number of grants that have been delayed is around 5 to 6 million total — maybe less, maybe only 3 million. But I know of a single grant that's 500K, and there are a few others. So somewhere between 3 and 10 million.
Austin Chen
And historically, the rounds themselves are around 20 million approximately?
Okay, so you saw this and thought maybe you want to do something better. And I imagine you also want to get other funders besides just Jan himself involved — having them participate, put in their funds, and help select recommenders, that kind of thing.
A lower-trust, more transparent funding process [5:04]
Oliver Habryka
That's right. What I really want is to make it so that if you're someone who wants to give away money — whether it's Dustin, or someone at the labs with a bunch of equity — there's a process they can join where they can observe what would happen if they were to give money to it. You can show up, see the applications, see the organizations, and ask: what would happen if I were to distribute $2 million through this process?
You get to see what the result would be — who you're deferring to, what they've said about relevant applications. Basically, at the last step, you get to decide how much money to distribute. That's a much lower-trust commitment than the default situation with a foundation, where you have to form an endowment, hire permanent staff, and build long-term relationships with organizations. I think that's a truly terrible experience if you want to make philanthropy easy.
Austin Chen
That's interesting — I actually wasn't even aware of how the SFF process works. And incidentally, I think I'm maybe one of the top 1,000 people in the world who cares about funding processes, and I still don't quite understand how it works.
Oliver Habryka
Their marketing has been really not very clear. Critch has been doing a good job with a few explanations in YouTube form, and they now have a demo app you can play around with. But a huge issue with the SFF process is just making the marketing clear and understandable — being clear about the value propositions, not just the mechanisms. Someone could derive the fact that you're allowed to go last as a funder and that this gives you a kind of assurance. But just telling people: hey, if you join this process, you get to know where your money goes before you spend it — in a way that's very difficult to achieve by almost any other means — that's the key message.
How the S-process works [6:53]
Austin Chen
Can you explain just a bit more about how that works? I had imagined the SFF process as: you pick some recommenders, give them the money, and the recommenders decide how to spend it.
Oliver Habryka
The key thing is that you work from the bottom up. You start with the applications. Then you have recommenders, and each recommender specifies a funding ordering over the organizations — essentially saying: here's the order in which I would fund these organizations. This is the one I'd give my first $10,000 to, then this one, then probably this third one. So they all specify a funding ordering.
The whole point is that they specify this ordering over a relatively wide range of money, because they might get different amounts flowing through them. And if other recommenders want to fund the same organizations, they want to ensure that if things get bumped further down their ordering, they can still reach that point.
Then the funders come in and decide in what order they want to distribute their money to the recommenders. At the end there's a final slider where the funder can say: do I want to put one million dollars into my funding recommendation over the recommenders, or two million, or three million, or zero? You can just see what grants get made if you give a marginal million dollars.
And because everyone has specified funding orderings, this allows you to just simulate what happens. This is where the name S-process comes from — it's a simulation process. You just simulate what would happen if you were to distribute an additional million dollars, and you get to see the counterfactual right there.
Austin Chen
Is the funder who's deciding whether to commit the next million dollars trying to figure out what allocation between the recommenders to give those million dollars to?
Oliver Habryka
They have some tools. Basically, you can decide which recommenders you want to give your next million dollars to, and then you can see what the recommenders would spend those million dollars on. You can choose how much and which different recommenders you're happy with if you distribute money through them. And I think it's very valuable to the system that the funder moves last.
The funder gets to look in and see who has done good reasoning. You want to really avoid certain kinds of gaming where people avoid ranking an organization highly because they expect other recommenders to recommend them first. So you want the funder to take some responsibility for checking whether anyone is gaming the system. It's really useful that the funder goes last — but this also means it's easy for the funder to pull out at the end and say, "I thought I was going to give $2 million, but seeing the quality of the recommenders' reasoning, I'm either pulling out completely or only giving $200,000."
Austin Chen
Maybe in my head the ideal thing I would try to design from first principles would be something more impact cert-like, or more standard VC equity-like, where the LPs are funding funds that then fund specific projects. And then every once in a while, the LPs — the Dustins and the Anthropic donors — decide, based on the performance of these funds or the specific set of charities they're funding, which ones they want to give money to.
In some ways, it seems pretty similar to the S-process as described.
Oliver Habryka
Yeah, maybe. Basically, you have a recommender, and I'm not at all attached to a recommender being a person as opposed to a team. Currently, recommenders are all individual people, but there are some weird edge cases. For instance, there exists an AI recommender that's been in every round, and there's a mean recommender, which just averages the implicit utility functions of everyone in the process — and that's an interesting thing you could decide to defer to that isn't necessarily a person.
Naming and communicating the value to funders [10:54]
Austin Chen
Where do you think were your biggest constraints on getting this built?
Oliver Habryka
The biggest thing right now is actually just trying to communicate what it is. What on earth does it mean to participate in the S-process? Generally, my biggest bottleneck right now is finding a good framing and name for it.
Is it a platform? Is it a market?
Austin Chen
I suggested "Exchange" earlier.
Oliver Habryka
I think "Exchange" really leans into the impact certificate type thing, which has some good attributes, but it's trying to cross a lot of inferential steps in one big pile. So trying to figure out a good communication that is centrally focused on helping funders — donors and philanthropists — understand what the product is.
I'm very happy about the Lightspeed Grants Round, which we started a while ago, but it's very oriented around communicating that it's going to be a good experience for the grantee. That's not my priority right now — I really want to get to a point where I can communicate the benefits of it to the funders.
Austin Chen
Yeah, and I have some sense that if you do the classic startup-y things — start with a small round, run more rounds down the line that kind of show the work — then people will be like, "Oh, cool, I can commit a small amount of money, or just commit virtual money to this to see what would happen, and participate without even having to commit anything except time."
Time, to be clear, is a big commitment. But maybe you could get people understanding this better.
Do you have a sense of when tentatively you might be able to do the first round?
Oliver Habryka
Right now I'm just writing up a Google Doc. I'm hoping to send it over to funders who might be interested in participating. I'm hoping to do something in August — that would be ideal. If we open applications a bit earlier, mid-July would be ideal for a public launch.
We could maybe even get a bunch of the applications that have gone to the Long-Term Future Fund and the SFF so we can start even earlier than that. Then having the first grant round dispersed in August would be ideal.
Austin Chen
And do you have a sense of how many dollars would be the sweet spot?
Oliver Habryka
For the first round, something like $10 million would be pretty good. It's a substantial amount of money that solidifies this as a real institution, but isn't going all in in some super intense way.
Austin Chen
And then probably from there, regular rounds — do you think like every half year?
Oliver Habryka
I think quarterly. There's a balance I'm currently uncertain about: what do you want the balance to be between the round evaluation mechanisms and speculation grantors? Right now the SFF runs once a year, but that's less bad than it sounds because the speculation grantors get to distribute a decent amount of money outside of the round, and then they get evaluated retroactively.
Austin Chen
That wasn't even obvious to me. It's not obvious that somebody applying right now would have a good chance of getting, say, $50,000 if they had a good project.
Oliver Habryka
Yeah, you'd just get it immediately via speculation grantors. So there's kind of a buffer there.
But I still think the annual process is really very slow and very heavy, and as a funder it's a pretty terrible experience because you only get one round a year to decide how to distribute the money. Currently I want to do quarterly, and then as things get crazier as a result of AI taking off and various other things, I'd expect to shorten that — eventually probably more like a monthly settlement, with a rolling application process.
As a funder, make it easy to copy over your preferences over recommenders from last round and adjust them a bit, as opposed to starting from scratch every time.
Austin Chen
One of my favorite sayings is: if it hurts, do it more. It's an agile kind of saying, usually around merging code. But if the funding process hurts and is painful, if you do it more, you'll be forced to do it better. More feedback.
Yeah, my current sense is that being a recommender is also quite painful because you have to read hundreds of applications.
Oliver Habryka
The most insane thing I absolutely don't understand is that the SFF really wants every recommender to read every application. And I'm just like, that is insane. Why would you do that? Most of these are random applications that nobody should have to read.
Recommenders should be able to specialize in certain domains. You could say, look, I don't know how to evaluate this weird biodefense stuff — I'm not going to look at those. I want to focus on a certain domain where I have expertise. It just doesn't make any sense that all of them need to evaluate all of them.
This is one of those procedural commitments that just makes running the S-process much, much harder.
Austin Chen
Yeah, I totally agree. That doesn't make any sense to me either.
Did you want to say anything about other funders, particularly Open Philanthropy, Longview, or Manifund?
EA philanthropy and the principal-agent problem: Open Philanthropy, Longview [15:51]
Oliver Habryka
I think philanthropy is one of the most cursed games in existence. You have people who, via entrepreneurship or sometimes investing, amass a huge amount of money that they've paid a lot of attention to — how they grew it, what they did with it. And then the usual thing is they're like, now I want to do something good with that money.
They don't have much ability — they don't want to pay that much attention to exactly what is happening with their money. They're usually busy with other projects. And now they're trying to hire someone to hand all of their wealth to, to distribute it to a set of projects that are hard to evaluate and where the metrics are much less legible than market feedback.
The principal-agent problem is just extremely difficult. The default outcome of what happens when rich people try to do philanthropy is that they think about starting a foundation. They imagine hiring someone on the market and ask themselves: who am I going to show up and feel comfortable trusting most of my net worth to? That doesn't make any sense.
So what they often end up doing is making a family office. The only way to solve this principal-agent problem is to choose someone where you have a very strong pre-existing relationship — someone you know isn't going to try to extract as much money as possible, or even just extract 5% of your money, which will turn out to be many millions of dollars. That trust bottleneck is just very intense.
I think GiveWell historically did something I thought was very good. Holden kind of built an institution where he said: you know you can distribute money through us because we're going to be extremely transparent. We're going to evaluate things in a way where you can follow our reasoning and see our receipts. We have this very intense commitment to giving money on the basis of cost-effectiveness — figuring out the most cost-effective grants that can help the world the most.
Over many years, they built a reputation of being an entity you can trust to distribute money on the basis of cost-effectiveness. And I think that's one of the things that enabled Dustin to trust them and put them in a position to distribute a much larger amount of money via Open Philanthropy and co-funding giving.
But as far as I can tell, they've now largely abandoned that as the central project. Now when you talk to them, they say a huge amount of what they're going to give away is stuff that puts them in a better position to direct future funds — stuff oriented around protecting and maintaining the brand and reputation of their funders.
Those things put you into much more of this principal-agent problem category — you no longer have a clear commitment to evaluating cost-effectiveness. You very rarely have any kind of inspectable public write-ups that people can cross-check. I think they're just going to be thrown into the midst of a lot of the classical principal-agent problems, and the forces of that are pretty messy.
Longview kind of never even tried. They're just trying to be an institution that somehow builds such a strong trust relationship with its funders that it can overcome the principal-agent problem easily, and I'm not that optimistic about it. Historically they've been extremely limited in what grants they can make because they really need to select from a subset of grants that look credible and legible in a way that just often gets rid of 99% of the impact.
There have been very few Longview grants I've been excited about historically.
Austin Chen
I mostly agree. I'm very confused why it seems like the Anthropic default — or among the set of Anthropic employees with liquidity in the current situation — is that they're planning on giving it mostly to Longview. I don't know.
Oliver Habryka
I mean, what's the alternative? That's the big question. Like, what else are they going to do?
But I also think that partially it's just kind of a con, in the sense that a lot of EAs are giving strong endorsements of Longview because they're worried that otherwise the money won't go to charity at all. So they're willing to say things like, "these are amazing advisors, these are really trustworthy people" — when I think the right answer is to say, hey, this is a really, really cursed problem.
There's a decent chance these people will be in a really, really difficult position where they'll be very tempted to recommend grants that sound good to you when you think about it only a tiny bit.
They can get more money from you, use that to establish more of a reputation, and there's this whole parallel game. These people are probably going to give you better advice than others who are also in this cursed problem class, but you should view them with a lot of suspicion — this is one of the most difficult principal-agent problems that exists. But that's just not what people are saying. People are saying these people are extremely trustworthy and great and you should give them your money. I think there's a thing going on where people want to make that recommendation because the money is very dead otherwise — it's not going to go to charity at all.
I'm currently expecting that a lot of Anthropic employees will be pretty burned in a year or two. When they experience these dynamics, it becomes very clear that a huge amount of grants are never presented to them because people are worried they'll find them weird. People will see that there's a lot of politics behind the scenes about which grants get put in front of them. People will see that there's a lot of weird reputational management — like, 'I can't make this grant because another funder would be angry at you if you made this grant.' That's really not what I want.
How much funding is coming [21:28]
Austin Chen
In some sense, I myself am not that worried about the way money plays out this year, and more worried about how funding plays out in the next year and the year after that.
Oliver Habryka
This year, I think everyone will struggle to just give money to anything, and a lot of things will be spinning out. I think we're going to see a huge amount of funding sitting in bank accounts — a lot committed, but very little actually distributed. Then in the following year and after that, I'm pretty worried about how that will play out, especially with an IPO perhaps.
Austin Chen
I heard some crazy numbers. Someone was saying maybe something like a hundred million dollars this year, then a hundred X that the next year, and a hundred X the year after that.
Oliver Habryka
The second 100X thing sounds tricky, but the first 100X thing sounds genuinely plausible to me. Yeah, 10 billion — that seems very plausible. 1 trillion, less plausible.
Austin Chen
Yeah, and then there are questions about how Anthropic itself, OpenAI, and other things play out. Maybe we can switch over and talk a little bit about incubating projects.
Surplus: the incubator [22:33]
Oliver Habryka
You'd like to see more interesting infrastructure and software that people with an altruistic mission in mind try to build to make the world a better place, and you've been thinking about running a fellowship of some kind.
Austin Chen
I'd maybe call it an incubator rather than a fellowship at this point — to be a little closer to Y Combinator than to MATS, perhaps, in terms of seniority or the kinds of outcomes we're hoping for.
The three main categories of things I'm interested in: one would be AI for epistemics and coordination — a broad class of things using LLM capabilities to help individuals, groups, or society as a whole reason better, think better, work together better. Second would be public-facing content, stuff like Asterisk, 80,000 Hours, or Blue Dot — sites that take ideas people in our community have thought about and try to disseminate them more widely. Third would be more internal-facing infrastructure — a better grants process would be an example, and historically things like Manifold and other internal coordination tools.
So those are three categories of projects I think I could help out a lot with. I'm open to other software projects and even non-software projects — I've done things like Manifest, which aren't very software-shaped.
Oliver Habryka
We made our conference largely as an app — though I think that's mostly not true. The people and the branding matters a lot. But I do think frequently, when I encountered a problem I'd seen at other conferences, I'd ask myself how I could solve it with software rather than just grit.
Austin Chen
That's a very good instinct to lean into, and especially easier now that vibe coding is so good. Some of my thesis about why to do this incubator now is that vibe coding is so good.
So many kinds of people who previously experienced problems can now just go hack and solve them themselves. One pointer to this is maybe Aella with Glosso.inc. In years past, I think Aella’s worked on other kinds of products that had some of the same ideas, but being able to execute on it herself makes it much better.
Why for-profits over nonprofits [24:37]
Oliver Habryka
One of the things that is most interesting about what you're trying to do is you're trying to do this project, but you're also trying to take equity in the organizations, or at least as much as that is possible. Have you figured out how to do that? Many of these people will maybe want to do it as a charity — are you going to encourage people to incorporate as an LLC, explicitly not as a charity? How are you thinking about structuring that right now?
Austin Chen
Yeah, and maybe explicitly as a C-corporation, not an LLC — whether a standard C-corporation or a PBC — just because those have better standard investor terms. I think some of it is just going to be stated upfront: we're looking for for-profits, for these reasons.
The reasons, very briefly, being that many kinds of for-profits encounter more feedback from reality because they're trying to get revenue and have to generate some value for the world, compared to many kinds of nonprofits that are research-oriented or vaguely community-building. I think it was a great thing when EAG started charging people for tickets, because it at least grounds some amount of value that EAG is providing.
A second reason to favor for-profits is the huge torrent of funding coming up. Some of these dollars will end up in nonprofits, but the nonprofits themselves are going to need a lot of tooling, infrastructure, and services that can be sold to them by for-profits. In that ecosystem, it will probably be better if orgs are charging each other what seems like fair value, as opposed to everyone just holding hands and giving everything away for free. Money helps solve coordination problems.
Something donors don't often appreciate about nonprofits — or 501(c)(3)s — is that 501(c)(3) dollars can go towards for-profits in order to buy services. And nonprofits can invest in for-profits. That's something Manifold actually does a lot — we often help donors put money in as a seed investment into a new for-profit.
Oliver Habryka
So are you planning for the entity that runs this to be a nonprofit or for-profit?
Austin Chen
Tentatively, the entity that runs this will be Manifund, so it will be a nonprofit, which means we can hopefully fundraise from 501(c)(3) donors. The standard thing Manifund does when putting investments into for-profits is that we hold money on behalf of the donor, like a standard donor-advised fund. And then if there's an exit scenario, the donor gets more money to invest into new charitable entities or other donations.
Oliver Habryka
That's kind of reasonable.
The ideal founder profile [27:11]
There are two things I'm most uncertain about for the project. One of them is: what is the kind of person you actually most want? It feels to me like there are three big cultural anchors that produce very different kinds of projects. You have the very impact-motivated people who are there because they want to do as much altruistic good as possible. You have the YC-type people you want to attract because they're mostly there for entrepreneurship. And then there are people who are just thinking, I can provide infrastructure to a bunch of the Anthropic people who will be extremely rich — I don't care about any of this impact stuff, but if I can get rich helping people, I'll gladly do it. And then you also have the rationalists who are thinking a lot about AI and epistemics, about how we can use these systems to reason better. Whenever I'm imagining this fellowship, I don't know where you're going to end up — this feels like one of the core tensions, and it will also determine how much synergy there is between the people and whether they'll actually successfully cooperate and gain value from each other.
Austin Chen
Yeah. I don't really have a good answer to that.
I kind of think there's a lot of things I respect about all three communities. I would try to market the incubator as much as I can to people in all these communities, and probably accept people who do well to be kind of like the intersection of some combination of these three communities. I'm hopeful that it's a large enough intersection that I can find at least 10 people who feel excited by startup-y things and rationalist-y things and impact EA-y things.
Different people will be more strongly tied into some of these communities. My current expectation is that I'll have a harder time finding good people from startup land to participate in this, maybe because YC offers more money than I have to offer. And YC is much more well-known — they're kind of like the Harvard.
I'm hoping I can attract some of them by being like, this is more like OG YC — intimate, small, a 10-person cohort with a lot of really good people. Currently, as a digression, YC does not get the best founders because the 200 or 500 person batches — however big YC is nowadays — feel more commoditized. Yeah, that's a good word for it. Like, a process.
So I'm really hoping that some of the best founders of the generation, who are even just not that AI-safety-pilled or whatever, look at this thing and find it kind of interesting.
Oliver Habryka
And also, in as much as you can pitch it as having a much higher density of interesting people — because with YC, you're asking how interesting is the average founder you're going to find — you can say, no, look, we're going for a much smaller, curated group. How many people are you currently imagining to participate in this cohort?
Austin Chen
Yeah, probably between 10 and 20 founders. So that would be more like 5 to 10 organizations, depending on the breakdown of solo founders versus co-founders. But that's my sense of what the original YC size was as well.
A lot of my constraints — or not necessarily constraints, but what I'm trying to do — is copy the homework of OG YC.
Whether writers can found startups in the vibe-coding era [30:26]
One question I was thinking might be interesting is: would Gwern, for example, be a good founder for this thing? He recently wrote this thing about guardian angels — it's kind of like digital twins, the idea of creating an LLM-shaped system that tries to be you as much as possible, as opposed to an assistant that is helpful for you. I think it's an interesting concept. I would be personally very excited if Gwern wanted to work on his idea in my incubator.
But yeah, someone like Gwern — that kind of writer-person — is that now a good fit for founding, because vibe coding is easy, et cetera?
Oliver Habryka
Oh, that's really interesting. There's been a joke going on ever since LLMs were even remotely able to code, which is: is the idea guy going to take over Silicon Valley? Like, maybe ideas are the only thing that matters. I don't currently believe it — you really need the iteration, you need to be able to talk to users, and there will still be tons and tons of grind work where a huge advantage will be the willingness to get through it. But we'll see how it goes.
Austin Chen
The other thing I would currently try to index for when looking for founders is people like me, people like you — people who have had the experience of building, but also are embedded in these communities.
Monetizing public communications projects [31:45]
Oliver Habryka
One of the things I least understand is the second category you mentioned about public communications — that one feels like the odd one out when I think about it. Like, how are you going to make money? I ran AI 2027 and we made zero dollars from it. Maybe we've been thinking about selling the movie rights — you could probably squeeze $100,000 out of those. It's a surprisingly popular media entity. But also, do you even want to?
Austin Chen
Yeah, I agree, it seems kind of hard. I mean, there's some ideal world where we have a new, great funder system that's happy to retroactively pay back people for doing good projects.
Oliver Habryka
In a world where you could potentially get a bunch of money from writing — which is proportional to the influence you get and the importance it plays in the world — that could be a huge shift and would make it more possible. But to ask people to work on a startup building that is kind of like building a bridge to nowhere, maybe, or maybe somewhere with a lot of gold, but it is very speculative.
Austin Chen
I do agree that the public comms category is the least monetizable in some sense, or at least the hardest to figure out how to get to revenue. There are also things about wanting to disseminate information freely. But there are some business models that do work — if you look at specific writers like Byrne Hobart, for example, he puts out a lot of free information and some amount of paid information.
Oliver Habryka
A Substack as a startup isn't crazy. Some Substacks are — we've never seen a billion-dollar Substack, but we are now in the range of seeing $10–20 million Substacks.
Austin Chen
I'm not quite sure yet about just being a Substack writer. Maybe we should just have really good Substack writers in the cohort. I don't really know — that's actually possible to me.
But also, if you look at things like 80K, I think there's room for something like 80K or Blue Dot to make money as a recruiting service. One of the major holes in the ecosystem — where there's lots of money flowing around — is finding good talent to hire when your org has a bunch of money but nobody to hire. I would encourage orgs like Blue Dot or Mats or 80K to think about whether, if you already have a pipeline and you're helping place people, you can ask the orgs to pay you every time you place a person.
Oliver Habryka
It's very tricky, because the whole value proposition of 80K is to tell you what the most impactful thing to do is — and suddenly you have this huge conflict of interest where maybe the most impactful thing doesn't pay you the most.
Austin Chen
Yeah, you don't want the failure mode of always placing people at Frontier Labs because those positions pay the most. I agree. But I remain open to it. You've got to capture some fraction of the value you provide somehow.
Oliver Habryka
That will always involve some amount of compromise and suboptimal incentives, because capturable value very rarely equals the full value you provide. But you still need to do it, otherwise you don't get to exist.
Austin Chen
Maybe a good thing to do would be to write down different funding models — how does Manifest make money, how does Lighthaven make money? I think those are two examples of models that actually have non-obvious business models but kind of work out, and it'd be worth exploring what goes into making those work. That's my hope for the second category, but fundamentally I agree — it's going to be hard to make a lot of money on those.
Oliver Habryka
You're currently imagining about two months for the program? Probably three months, starting late July.
Austin Chen
I was about to launch today, but then the Claude or Mythos team dropped something and I thought it was a bad day to launch — maybe I'll try tomorrow. Bad day to launch — I've got to redo my launch website with Mythos now. It's like the cheapest alpha just sitting on the ground right now.
Oliver Habryka
I'm curious if you had any thoughts on things you would personally look for if you were joining an incubator. I gather this isn't how you would do it yourself.
The thing I've always wanted out of YC, and that seemed most interesting to me, is that it's very clear my organization routinely faces extremely important decisions. I have friends and other people to talk to, but it seems to me that many of these problems are ones that many other people have faced before — core decisions like wanting to scale up, wanting to hire a lot, and how to do that without destroying your organization.
I would genuinely be quite interested in talking to a bunch of people who have been through this, who have dealt with VC stuff, to help me understand how they handle negotiations with funders and things like that. There are a lot of norms — or rather, just practical matters — around how these negotiations work that seem very helpful to understand.
High-stakes business advice has felt like a thing that's just very hard to get otherwise. Maybe I can post more frequently on Twitter, but often this is stuff that's just very hard to talk about publicly and hard to navigate. That's one of the things that has always attracted me most to YC — just having access to people who've seen a bunch of stuff in business, so I can go to them and say,
"Hey, I have this huge conflict between two of my employees that's threatening to tear apart the organization." That's never happened to me, but I've seen it happen to other organizations. Getting advice from people who have navigated those kinds of really tricky situations is incredibly valuable.
Austin Chen
That's really interesting. I kind of see you as a big fish in a small pond, where lots of people currently go to you for advice. I remember recently somebody on the board of Manifold asked you about a related question. I hadn't thought about it, but it totally makes sense that you'd want to get advice from people who have seen even bigger scales of problems.
Oliver Habryka
As a CEO, it's pretty common for CEOs to have other CEO friends whose relationships are comparably strong to those within their own organization, just because CEOs face similar problems. It's very hard for someone to understand the kinds of issues a CEO faces if they've never faced them themselves. Having more access to a community of CEOs seems very valuable to me.
The vast majority of people I get to talk with are just researchers. I know maybe five CEOs — it's not that many.
Austin Chen
I kind of wonder if there are enough sizeable organizations in the AI safety space that you could pull together, like Beth Barnes and the Apollo people or something, and get answers to some of those questions. Though I feel like the research orgs aren't quite right for what you have in mind.
Oliver Habryka
That's right. I frequently have a sense that something doesn't feel quite right. I do believe that Silicon Valley has successfully developed a real lineage of management knowledge. If I went through YC, there are people I could talk to who have been partners and have been thinking about these problems for two to three decades.
A lot of people have sampled from 100 to 200 perspectives over that time. I'm very confident that almost everyone around me is just making it up. Very few people are really tapping into a longer lineage. This is obvious to me because I'm a huge fan of the Toyota Method and Lean Startup type thinking.
There's not a single other CEO around us who feels to me like they really get it. I don't think Buck gets it, I don't feel like Beth gets Lean Startup stuff at all. I just look around and think maybe people have different management philosophies, but I would really like to tap into a lineage more exposed to that kind of thinking, because I think it's the most successful lineage in the history of humanity for creating successful organizations.
Austin Chen
One person that comes to mind is Steve Newman. I don't know if you've met him.
Oliver Habryka
Oh, that makes so much sense. Somehow Steve Newman feels so chill.
Austin Chen
He is — he's just good. And it's top of mind because recently he was working with my wife Rachel on the Golden Gate Institute. Steve and also Taren, who's the CEO of that org, are both very excited about AI safety related things and have a lot of experience.
Oliver Habryka
Yeah, that sounds great — I should talk to them more. And I think more people are going to be like that, having worked at high-powered organizations and now wanting to do something about AI. I'm pretty excited about pulling together a network like that.
It's going to be hard to compete with people who have been YC partners for 10 years, but it's not implausible that when I look at the first cohort, everyone who's building something there could actually be a good peer in navigating these situations.
I hope that you get many first-time founders, but also getting second- to third-time founders — I think that's a real thing.
Austin Chen
I don't quite know currently whether I should be angling for more first-time or more second-time founders.
I implicitly feel like the terms I can offer are going to be embarrassing — quite tricky. A second-time founder who sold a $100 million organization might ask, do I really want to take $100K for 10% of my company?
Oliver Habryka
I wonder whether that's a strong argument for doing more variable terms. It has of course helped YC enormously to have standardized terms, but it's plausible that you're in a more competitive market. Because the early singularity startup founder market will include a lot more high-powered people, there will be many in strong negotiation positions. So for some of them, you might just want to offer much better terms — though it seems pretty tricky to navigate socially and puts more adversarial pressure on you.
Austin Chen
I really hate negotiating, but I'll think about it. I was hoping for just one standard term. But YC does, for very prestigious or third-time founders, allow for different terms — they just don't advertise it. Of course they don't want the word getting out.
Nobody at YC has told me this, so this is speculation.
One topic I'm interested to hear your thoughts on is: why work on funding something versus incubating something versus directly doing the thing yourself?
Oliver's case for the incubator [42:09]
Oliver Habryka
I'm tentatively excited about this project for you, which is actually a huge bar — because for years I've been on a crusade against incubators. I've advised maybe two dozen people to please not start an incubator.
The usual reason I say that is because they're trying to incubate in a domain where they've never produced any valuable work themselves. I think of things like the FLI fellowship — they were trying to get people to do interesting AI-for-epistemics work, but had never done any themselves. Similarly, I've seen research organizations want to build an incubator for good research when they'd never produced any interesting research themselves. I just don't think that's a good idea.
This was in some sense the central secret sauce of YC — it was an incubator by founders who had experience founding things. You should only ever consider starting an incubator after you've actually started one instance of the kind of thing you want to incubate. So you've got to start with building something yourself.
I feel slightly better about being a funder rather than an incubator — though it still seems very risky. Can you fund work you've never done yourself? It's tricky, but slightly less doomed, because you have a much less intense effect on the culture of an organization and you're not trying to tell people how to run things. You can still become a huge bottleneck, but it's slightly more reasonable to figure out how to make a binary funding decision using deference chains and asking others for advice.
On professional grantmakers [44:04]
I still wouldn't become a funder if I'd never done good work in the space. The only reason I feel comfortable moving into funding is that I've always advocated for funders who also have a double identity. One of the things I'm most excited about for both the existing process and anything I want to build is having many recommenders who actually work in the space and are actively connected to it.
I don't really believe in professional grantmakers as a role that benefits from specialization — I think it's deeply harmed by the fact that you don't get to experience things. So this is a pretty big critique of the whole CG model.
Yeah, I don't think professional grantmaker is the right role. The only thing that has historically kept CG on rails is that many of the people there actively published their ideas — they were part-time researchers and part-time grantmakers. I think that was a balance that could work. But CG has completely stopped doing that.
Nobody at CG runs the kind of blog that Holden Karnofsky used to write or Joe Carlsmith used to write and stuff like that.
Austin Chen
Yeah, I mostly agree — this is why I still like re-granting a lot. The concept of re-granting is that you give money to people who are very part-time grantmakers and mostly doing their research, field building, or whatever they're really good at. And then if they find good opportunities, they can apply money towards that, while still being grounded in the actual work.
Oliver Habryka
So I guess, answering your question about how to trade off — just go and build something that actually provides value that people would be willing to pay for. Even in a nonprofit space, try to do something where you find a customer, whether that means you can clearly demonstrate impact, sell a concrete product, or charge admission rates or anything like that.
After that, I would consider doing something in the incubation or funder space. The multipliers are enormous — it's obviously the case that YC has had a much bigger impact. It's very hard for me to imagine Paul Graham doing something that would have had as much effect on the world as starting YC. It has shaped Silicon Valley beyond what I think any startup could have realistically done.
So obviously, often it's the right choice to work in infrastructure and things like that.
Austin Chen
Yeah, but I often get these — I don't know if you've been getting them — very young undergrad people who have recently seen something like Nan Ranselhoff's article on third-wave philanthropy and thought, "Oh yeah, there's going to be a lot of money, I should do something about this." And I'm like, you might not be the person who is well-placed to do something about this.
Oliver Habryka
In some sense, I think they're well-placed to do things that other people might want to pay for. But they shouldn't try to start a private fund — that doesn't really seem like the right response to this situation.
Austin Chen
Yeah, I mostly think — and I'm kind of talking my book here — that these people should start orgs or projects that can take funding if the projects do well. Because the scarce thing when there's a lot of money is orgs, projects, people, and good ideas. Doing that directly seems like more good than doing meta-level stuff.
Whether infrastructure work is more direct than safety research [47:36]
And that consideration applies even for you and me — we can do infrastructure-level meta stuff, or we can just directly do a specific thing that seems really good. I don't know if you try to think about how to navigate that personally.
Oliver Habryka
Especially for existential risk — there's a straw man I think people have that I really don't get, where they talk about "direct work on x-risk." I'm like, what do you mean? I don't know what direct work on AI safety is. Do you mean research?
I think the work I do has a much shorter causal chain to having good AI outcomes than people who are trying to solve mathematical problems adjacent to AI safety — because they then need to tell a story. The story for the vast majority of AI safety researchers routes through the existence of platforms like LessWrong. How can LessWrong have a longer causal chain than the research itself? The research would have nowhere to go without an active field.
The research would have nowhere to go without these things. These are just causal chains that I think are ultimately actually shorter than the vast majority of what people conceptualize as direct work. AI safety and AI existential risk is the kind of cursed problem where it's very hard to have anything that feels like directly solving the problem.
Sometimes people imagine that if they're at a lab, their impact can come from directly solving the problem. But that's really not the case. The vast majority of people at a lab will find that their impact comes from the degree to which they're creating a culture that is less likely to deceive themselves about the difficulty of the alignment problem, which will enable advocacy for policy changes at a crucial moment in time, which requires other people to be listening at the right time, to then be proposing the bill that can get enacted so we can get international cooperation to solve it.
And you're like, yes, that's the same long causal chain as everyone else. I think people are so attracted to this idea that they rationalize a huge amount of work as being useful when I think it's basically useless.
There's a huge amount of very local alignment research where people say, "I made the alignment benchmark go up, that must mean I helped the problem, I solved the problem." I really don't think the action is in the alignment benchmarks. The alignment benchmarks aren't tracking the problem well enough.
That's really not where I think almost any of the likely failures of this whole situation will come from. I agree it can feel concrete and specific, but as much as you're hoping to have your eyes on the ball, I don't think there's a sense in which it's more direct to work on a problem.
Austin Chen
I had a very toy model of, oh yeah, the technical AI safety researchers, that's direct work. And then the field builders, like you and me, we're doing the meta work of helping these people. But yeah, I can see it.
Oliver Habryka
No, the safety researchers rely on institutions like you and me to exist to even do their work. So at the very least, we're at the same level of abstraction. And I would say that I'm placing myself much, much closer in terms of uncertainty.
When I'm doing things like improving the quality of all the AI safety research at once, I get to take distributed bets, I get to make changes that pay off much more quickly than the average safety researcher. When I set up changes about how content gets popular on LessWrong or the Alignment Forum, or various UI improvements, I get to see the differences in how many people read that stuff and where its impact is at a scale of weeks relatively quickly.
A researcher will release a report every six months to two years, and they need to rely on there being infrastructure to publish that kind of stuff.
The case for a better AI safety journal [51:08]
Austin Chen
Kind of on that note, one of the ideas floating in the back of my head is a better AI safety journal or something like that. I think the Alignment Forum is reasonable and LessWrong is very good, but neither of these things are quite the thing yet.
Oliver Habryka
I'm super excited about that. If I had more time, it would be at the top of my list of projects I would want to do.
LessWrong unambiguously has post-publication peer review in the sense that you publish your stuff and then everyone will tear it apart in the comments. It's much better at that than the vast majority of ML and AI academia peer review. The post-publication peer review is genuinely high quality and tracks quality to a really substantial degree.
But the key thing is that right now it relies on a relatively fragile social status game where you can get reputation by publishing highly upvoted comments, but the visibility of comments is kind of mixed. You are building reputation, but only among a relatively small crowd who can really evaluate whether your comments are good. Having something where we're paying people money to do reviews of good work would be a big improvement.
That might be a lot of money, because reviewing good work is actually really valuable. I would be hesitant to have people specialize as reviewers too much — going all the way to being a full-time reviewer would be an issue similar to my concerns about starting an incubator or funder. But getting to a point where you're a half-time reviewer and see it as a major part of your professional responsibility to review other people's work is definitely where I want to get, and where the current field is really falling quite flat.
Yeah, I'd love something in that space. I've thought of a bunch of cool mechanism designs and ways to do stuff like that. But at the end of the day, it needs a lot of taste and a decent amount of communication and branding. And I'm just fully booked out.
Austin Chen
The first one that comes to mind as someone who's kind of doing this thing is Zvi. A lot of what Zvi is doing is summarizing a bunch of things and giving his takes, but also giving other people's takes.
Oliver Habryka
I loved Rohin Shah's alignment newsletter — it was great at the time. He ran it for about three years before he went to DeepMind.
Mutual feedback [53:17]
Austin Chen
What's one thing you admire about me and one thing you would criticize about me? And I'll do the reverse.
Oliver Habryka
Building good infrastructure is very hard. I think both Manifund and Manifold are just two great pieces of infrastructure that I'm very happy exist.
You're one of the few people I talk to who gets things that feel very basic to me — like PR FAQs, getting feedback quickly, failing fast, doing painful things more frequently. A lot of ideas around putting yourself in touch with reality as much as possible at every step to find whatever gradient is there. You feel like one of the few people who get that.
While also doing a pretty good job at keeping the big picture in mind. Most people in Silicon Valley, when I talk to them, take that mindset and just apply it to starting a company. They don't really apply it to asking what the most important problems in the world are.
My biggest criticism of you is that I think there's something you're missing about being careful around people. You're very high on my list of, like... "God, I bet Austin is going to like this crazy sociopath.” I don't know exactly what it is, but I think it might be related to how you relate to the company stuff.
I've been trying really hard to build a community with norms that maintain a certain kind of higher-trust relationship. I think you are trying to do something principled — you've aggregated principles around radical transparency and various things like that.
But I have long modeled the world such that if you're trying to shape the trajectory of AI, a lot of your effort should go into figuring out whether you're helping or hurting. If you don't assign substantial probability that you're making things worse, you're probably very wrong. It's a domain where the considerations are tricky and the right ethical principles are crucially important.
I think you have a very deep distaste for a certain kind of scrupulosity — where people obsess about whether they're doing good or causing harm in the world. I'm sympathetic to that, but I think you're not seeing a version of it that I actually think is good.
It is eminently possible to think about your role in the world, realize that you have caused grave amounts of harm, and use that to learn better — and to have a culture of people who take that question really seriously. I expect that if you keep building lots and lots of infrastructure, there will end up being a lot of people who make the world worse, and you will fail to have a culture that prevents that.
I think there's an underlying thing here where it's very, very tricky to have an ecosystem that does credit accounting in a way that is sensitive to whether you made the world better or worse, if you already have a substantial number of people who would be threatened by that becoming a fact about the ecosystem.
If you're in a world where there's a bunch of people who have made the world a lot worse — because they started frontier companies and really didn't try very hard to be careful — those people will form a very powerful coalition to prevent anything that could meaningfully create accountability. And so I'm definitely worried that in starting an incubator like this, there's going to be a bunch of people who make the world a lot worse, and it's going to make it very hard to have open conversations about that, because those people will very rationally understand that their standing in the ecosystem is going to be threatened.
And that's going to be very tricky.
Austin Chen
Thank you. I very much appreciate that.
I think you have many traits which I admire. One that I'm going to call out, that I think people don't notice as much, is that I think of you as being very service-oriented, really trying to help people. You often get in the weeds of trying to debug problems or fix things personally.
I was asking Lauren the other day who has stayed in the most rooms at Light Haven, and she said it's probably you — you do this all the time, you kind of dog-food them. I think the dog-fooding thing is related, and part of it is that you care a lot about making things really good in the service of serving people. I think it's a beautiful attitude and I would like to see more people adopt it.
Oliver Habryka
I'm glad. Now hit me.
Austin Chen
I think the thing where you got into a fight with Scott Alexander this year — that seemed really dumb. Maybe you kind of agree.
Oliver Habryka
Here's the thing you need to understand. If you start endorsing destroying the world, something really bad has gone on. You need to maintain a culture in which people can talk about this stuff. And if somebody is like, "How dare you attack these people who are carrying my banner? Aren't they trying to do good? How dare you even consider that maybe they're making the world a lot worse?" — I'm just like, no.
I cannot have a culture around me that is trying to protect the dignity of powerful people just because they might be scared to contribute. If we want to be morally serious, we need to maintain a strict standard. But I agree — it's tricky. I don't know how to navigate this stuff. I'm very confident that I'll look back at most of what I've done over the last few years in this domain and think I was an idiot. The problem is I just don't know which direction it'll come from. There's a lot of thrashing around when I'm trying to navigate a situation.
I'm definitely not being consistent in a way where I feel like I could behave this way if I had gotten the right answer. But I think I'm getting feedback about which things work, what the consequences of various attitudes in the space are, and I expect I'll figure it out and have slightly less dramatic public fights. Though maybe not — public fights are a pretty good way of working things out.
Austin Chen
Some of my favorite comments or writing from you have been public fights, like the coefficient giving thing. It really helped clarify my understanding of the space.
At the same time, I think it's probably good for you, your reputation, and the projects you care about to not be known even more as someone who fights a lot.
How to help: LessWrong, Surplus, and the S-process [1:01:01]
Maybe let's end on a classic podcast question: how can people help you out with the products that you're excited about? Either the S-Process thing or something else.
Oliver Habryka
A big one is we're finally in a position of financial stability to hire. Historically we've been the slowest growing organization of anything in the space, but I think we might want to hire three people this year. Three people. And as much as anyone watching this could be you, possibly — yeah, that's the big one.
Always contribute to LessWrong, contribute to the intellectual commons. I think that is at the margin very, very highly undersupplied still.
For the S-Process work — if you're a funder who has been intrigued by anything I've been saying about making grant-making more trustless, making it possible to get access to a huge amount of deal flow, now is a great time not only to potentially participate in the process, but also to shape it. If you reach out to me now and say you might be interested in distributing some money, you might be one of those early dogfood users where I build a product around you because you're my one source of feedback about how to do something good.
Austin Chen
And I think that's often a quite amazing experience. I can attest to that from my own experience, both with Lighthaven for Manifest the first time around, and more recently Waypoint for Manifest again.
Oliver Habryka
We have a gradient to optimize on, and that gradient might be you.
Austin Chen
I'll answer my own question too. Some things we're looking for: first and foremost, just great founders. Either if you yourself are interested, or you know people, or you're willing to help promote this — we should be launching very soon, so let people know Surplus is a place you can go.
Oliver Habryka
You never actually said "Surplus" in this whole session! Yes, it is called Surplus — that's the name of the incubator. And surplus.dev was the URL you were considering, which will probably be the official URL.
Austin Chen
We are also raising funding for Surplus. Doing the math, it's about 100K per founder, so maybe one to two million dollars total. Not that much in the grand scheme of things. I don't have any of it raised yet — I'm going to empty my coffers to make this thing work if I have to, but I'd prefer not to.
Oliver Habryka
In as much as you're open to donor-advised-fund-style arrangements, someone could come in with a million dollars and have essentially a Manifund stake with equity they could potentially do something with. Right — it could be equity, it could be a loan, depending on what the funders are looking for.
Austin Chen
I'm hoping that with enough money flowing around the ecosystem, somebody will look at this and think, yeah, this is exciting — I'm willing to pitch in a few hundred thousand dollars or something like that.
The third thing is that some people listening might be the kinds of speakers I'd be excited to bring in. You, Oli, would be a great person to bring in at some point. But also others who've started similar projects and maybe don't want to commit a lot of their time, but would come in and give a talk — those are the people I'm looking for.
And then general hiring as well — we're looking to hire around three people pretty soon for the combination of Manifund, Mox, and Manifest. I think the people who would be good working at Lightcone might also be good here.
Thank you very much. This was a lot of fun.