I am an evaluator for Lightcone Commons (LC). LC is a platform for applicants to submit projects, for evaluators to rank them and for funders to choose the funding orders of several evaluators. It is not a foundation, in that it doesn’t have money that it spends how it chooses. And I do not have a permanent position working for Lightcone Commons. I get paid 2% of any money that a funder chooses to direct via my funding order. This pays for my time and also for any research that I pay others to do.
That said, what were my general and specific funding preferences?
This is a version of what Lightcone published, it might be wrong, but theirs was definitely wrong (their headline numbers didn’t include funding that had been hidden)
Lightcone also has a beautiful page for looking through public information here.
General preferences:
I like AI policy that creates the ability to slow down AI if necessary, but doesn’t automatically push for immediate pauses. I like policy orgs which people I know in US politics respect. My current understanding is that it’s organisations that work with the administration, are respected by Republicans or that the AI companies listen to. Bipartisan organisations, I am told, are considered Dem. It’s not worth talking to random Senators, it should be those who tend to push/change legislation that gets passed. This is true even with a Dem majority. Policy will still need Trump’s non-veto and Republican votes.
I flag conflicts of interest but still recommend those orgs if they’re good. I don’t recommend myself or my company. Beyond that, I fund the work I think is best.
I’m less confident of AI safety work than the median evaluator. This is both because I tend to think AI safety work sometimes subsidises AI research and because I don’t know what good AI safety work looks like. I funded a few pieces of AI safety work that other evaluators said was valuable to them (notably Anima).
I like things that have incentive alignment, but might not get VC funding initially. Some version of Pangram that is trying to get on its feet. In practice there wasn’t much like this.
I rate epistemics/competent-small-operator projects more highly than the median evaluator. I want things which make trust and the information environment better and not worse. Overall, I wanted few organisations that I thought might damage trust, and some that would build it.
A quick caveat. When I talk about “what I recommended”, I am going to talk about dollars that moved according to my recommendations. About $5m was funded according to my recommendations (for which I was paid $100k). However because other people also recommended those things (and because some things were vetoed by funders) the final $ that moved according to my preferences was $11.2m down my list. So when I say ‘my dollars’ I mean that the dollars of recommendation that are allocated via me, which might be much smaller than the total amount a given organisation received.
What I recommended in the part that was funded, by type1.
By dollars (roughly): Policy 53%, epistemics 26%, Advocacy 17%, AI safety work 4%
In terms of conflicts of interest, about 30- 40% (~$2M of $5.2M) of my dollars went to people I am close with or have some clear conflict of interest. I flagged these to funders
Some organisations (mainly relating to policy) wished to remain private, so I have removed them. Specific rationales for redacted entries:
• As above I generally think AI is very important and reducing risk without necessarily leading to an immediate pause is a high priority. That’s true of several of the private entries, that seemed common sense risk reduction and were trusted by politics people I trust. If you are a wealthy person who can give to Republicans I have more recommendations.
• Then there are some lobbying orgs that seemed very good value for money that lobbied on a range of topics, run by people I trust.
Otherwise this is my funding order:
Again, not all this is ‘my’ ‘money’ since often these were partially filled up by others who agreed with my recommendations.
Rationales:
Sparrow infrastructure ( $1.24M in my order) - I like Lighthaven and for an (unaffiliated) attempt at ‘Lighthaven East’ core funding seems cheap. I talked to some political people and they seemed to think that having a space to bring AI safety people and political people together was a good idea. Rationality meetups ( $700k) - I like Skylar a lot and respect his judgement and ability to manage people. Rationality has been valuable in my life and if, I suspect, this whole AI thing goes on for a while longer than people expect I would like more people who come into contact with notions of having ideas that can be proven wrong, trying to actually get to the right answer, holding things lightly etc.
AI Impacts ( $700k) - I think Katja was deeply prescient in getting AI surveys going in the mid-2010s. And I, and I think many other people find her work illuminating and her a useful person to discuss AI with. Yes, she’s my ex, which I flagged, but if you look through the kinds of people she has worked alongside, I think she has been an underrated partner in a significant part of AI discourse over the last decade.
Manifold Markets ($100k) - I often use Manifold while working to sense check various forecasts and I like to support infrastructure. Unclear how that will continue going forward, with AI forecasting, but let’s see. I recommended this donation, but that everything over $150k go as an investment.
Technology Strategy Roleplay ($300k) - I tend to think that roleplay is a good way to get people into the mindset of making decisions. I wish politicians roleplayed a lot more to get a sense of some obviously bad decisions to avoid. This team is well respected.
80,000 Hours, the video team specifically ( $1.16M) - I think that Chana and her team manage to talk honestly about AI risk, trying to acknowledge the caveats. On the Sam Altman video they took a subject I had looked into quite deeply and told me new things while not getting anything wrong I’d heard from elsewhere. I want more output from them.
Anima Labs ( $300k) - People I trust said that it helped them make sense of AI while not seeming like work that was subsidising the AI companies.
AI Wow ($50k) - I found Peter pretty inspirational when I saw his workflows. I would like more people speeding up orgs in the AI safety space and $50k seems like fine to get a person started (though in future I’d like him to be paid by organisations).
Sentinel research ($250k) - I like nuno’s horizon scanning work and often use it in my own. I would like to see more from him.
MIT AI Research RCT ($100k) - it seemed sensical. I cannot remember the full details.
Swift Centre ( $150k) - Sometimes there are people who want forecasts who are unable to pay for them. I would like the swift centre (who I work for very part time) to be able to offer these.
I want to note briefly how frank I am being here. Please disagree with me and I hope in future rounds I will feel my rationales are more robust, but this was the work and I did it. I am not going to pretend I always had better reasons than I say here (though sometimes I did, and can’t say) and sometimes I will just have more to say. If you think something was ludicrous or reasoning bad, please say so.
I think that’s the meat of the discussion, but here are some other comments:
How I differ from the other evaluators
Eyeballing, my funding was the second-most uncorrelated with the main picks.
There were a lot of big consensus picks I didn’t go for. If political, it was because they seemed to be talking to the wrong politicians (not those who would likely get votes through). If advocacy because I couldn’t trust them to make things better not worse, though I worry I was too cautious there. I am interested in finding good-not-bad advocacy orgs this time.
I was surprised that $5m was routed by me and have been told that it was probably because my list looked so different to others. I wasn’t sure how much money I was going to get from this, nor is this my full-time job. I think I would have spent more time on it if I had realised $5m was going to be allocated through me. I think had I done so perhaps I would have had even more go via me.
It is a bit strange to be someone who can so dramatically affect the lives of ones friends but also feels strongly motivated to ‘say’ true things. I expect this to only get stranger in future rounds. I enjoy people thinking I am worth arguing with and convincing, though it’s hard having to rate projects lowly that one knows.
Please apply!
If I can give one piece of advice, please apply if you have an effective project. I want good projects to recommend at the top of my order.
I intend to write another article about what a good project and application looks like, but in short, I am interested in projects that focus on:
Liability
Advocacy orgs I can trust
Effective lobbying in anywhere
Improve epistemics generally (watch this space)
Things that will improve governance in countries/states
And a good application looks like:
Why should I care
What is a simple description of why you should get a marginal dollar
Who knows of your project and respects it
But I’ll write these more carefully in another blog. Apply!
Also I am leading an epistemics fund, sponsored by Ought. You can read more about that here and I’d appreciate retweets when I write about it tomorrow morning:
I am an evaluator for Lightcone Commons (LC). LC is a platform for applicants to submit projects, for evaluators to rank them and for funders to choose the funding orders of several evaluators. It is not a foundation, in that it doesn’t have money that it spends how it chooses. And I do not have a permanent position working for Lightcone Commons. I get paid 2% of any money that a funder chooses to direct via my funding order. This pays for my time and also for any research that I pay others to do.
That said, what were my general and specific funding preferences?
This is a version of what Lightcone published, it might be wrong, but theirs was definitely wrong (their headline numbers didn’t include funding that had been hidden)
Lightcone also has a beautiful page for looking through public information here.
General preferences:
A quick caveat. When I talk about “what I recommended”, I am going to talk about dollars that moved according to my recommendations. About $5m was funded according to my recommendations (for which I was paid $100k). However because other people also recommended those things (and because some things were vetoed by funders) the final $ that moved according to my preferences was $11.2m down my list. So when I say ‘my dollars’ I mean that the dollars of recommendation that are allocated via me, which might be much smaller than the total amount a given organisation received.
What I recommended in the part that was funded, by type1.
Some organisations (mainly relating to policy) wished to remain private, so I have removed them. Specific rationales for redacted entries:
• As above I generally think AI is very important and reducing risk without necessarily leading to an immediate pause is a high priority. That’s true of several of the private entries, that seemed common sense risk reduction and were trusted by politics people I trust. If you are a wealthy person who can give to Republicans I have more recommendations.
• Then there are some lobbying orgs that seemed very good value for money that lobbied on a range of topics, run by people I trust.
Otherwise this is my funding order:
Again, not all this is ‘my’ ‘money’ since often these were partially filled up by others who agreed with my recommendations.
Rationales:
I want to note briefly how frank I am being here. Please disagree with me and I hope in future rounds I will feel my rationales are more robust, but this was the work and I did it. I am not going to pretend I always had better reasons than I say here (though sometimes I did, and can’t say) and sometimes I will just have more to say. If you think something was ludicrous or reasoning bad, please say so.
I think that’s the meat of the discussion, but here are some other comments:
How I differ from the other evaluators
I was surprised that $5m was routed by me and have been told that it was probably because my list looked so different to others. I wasn’t sure how much money I was going to get from this, nor is this my full-time job. I think I would have spent more time on it if I had realised $5m was going to be allocated through me. I think had I done so perhaps I would have had even more go via me.
It is a bit strange to be someone who can so dramatically affect the lives of ones friends but also feels strongly motivated to ‘say’ true things. I expect this to only get stranger in future rounds. I enjoy people thinking I am worth arguing with and convincing, though it’s hard having to rate projects lowly that one knows.
Please apply!
If I can give one piece of advice, please apply if you have an effective project. I want good projects to recommend at the top of my order.
I intend to write another article about what a good project and application looks like, but in short, I am interested in projects that focus on:
And a good application looks like:
But I’ll write these more carefully in another blog. Apply!
https://www.lightconecommons.com/apply
Also I am leading an epistemics fund, sponsored by Ought. You can read more about that here and I’d appreciate retweets when I write about it tomorrow morning:
https://epistemic.fund/