On Wholesomeness

Wiki Contributions



I think that you're right that people's jobs are a significant thing driving the difference here (thanks), but I'd guess that the bigger impact of jobs is via jobs --> culture than via jobs --> individual decisions. This impression is based on a sense of "when visiting Constellation, I feel less pull to engage in the open-ended idea exploration vs at FHI", as well as "at FHI, I think people whose main job was something else would still not-infrequently spend some time engaging with the big open questions of the day".

I might be wrong about that ¯\_(ツ)_/¯


I feel awkward about trying to offer examples because (1) I'm often bad at that when on the spot, and (2) I don't want people to over-index on particular ones I give. I'd be happy to offer thoughts on putative examples, if you wanted (while being clear that the judges will all ultimately assess things as seem best to them). 

Will probably respond to emails on entries (which might be to decline to comment on aspects of it).


I don't really disagree with anything you're saying here, and am left with confusion about what your confusion is about (like it seemed like you were offering it as examples of disagreement?).


(Caveat: it's been a while since I've visited Constellation, so if things have changed recently I may be out of touch.)

I'm not sure that Constellation should be doing anything differently. I think there's a spectrum of how much your culture is like blue-skies thinking vs highly prioritized on the most important things. I think that FHI was more towards the first end of this spectrum, and Constellation is more towards the latter. I think that there are a lot of good things that come with being further in that direction, but I do think it means you're less likely to produce very novel ideas.

To illustrate via caricatures in a made-up example: say someone turned up in one of the offices and said "OK here's a model I've been developing of how aliens might build AGI". I think the vibe in Constellation would trend towards people are interested to chat about it for fifteen minutes at lunch (questions a mix of the treating-it-as-a-game and the pointed but-how-will-this-help-us), and then say they've got work they've got to get back to. I think the vibe in FHI would have trended more towards people treat it as a serious question (assuming there's something interesting to the model), and it generates an impromptu 3-hour conversation at a whiteboard with four people fleshing out details and variations, which ends with someone volunteering to send round a first draft of a paper. I also think Constellation is further in the direction of being bought into some common assumptions than FHI was; e.g. it would seem to me more culturally legit to start a conversation questioning whether AI risk was real at FHI than Constellation.

I kind of think there's something valuable about the Constellation culture on this one, and I don't want to just replace it with the FHI one. But I think there's something important and valuable about the FHI thing which I'd love to see existing in some more places.

(In the process of writing this comment it occurred to me that Constellation could perhaps decide to have some common spaces which try to be more FHI-like, while trying not to change the rest. Honestly I think this is a little hard without giving that subspace a strong distinct identity. It's possible they should do that; my immediate take now that I've thought to pose the question is that I'm confused about it.)


I completely agree that Oliver is a great fit for leading on research infrastructure (and the default thing I was imagining was that he would run the institute; although it's possible it would be even better if he could arrange to be number two with a strong professional lead, giving him more freedom to focus attention on new initiatives within the institute, that isn't where I'd start). But I was specifically talking about the "research lead" role. By default I'd guess people in this role would report to the head of the institute, but also have a lot of intellectual freedom. (It might not even be a formal role; I think sometimes "star researchers" might do a lot of this work without it being formalized, but it still seems super important for someone to be doing.) I don't feel like Oliver's track record blows me away on any of the three subdimensions I named there, and your examples of successes at research infrastructure don't speak to it. This is compatible with him being stronger than I guess, because he hasn't tried in earnest at the things I'm pointing to. (I'm including some adjustment for this, but perhaps I'm undershooting. On the other hand I'd also expect him to level up at it faster if he's working on it in conjunction with people with strong track records.)

I think it's obvious that you want some beacon function (to make it an attractive option for people with strong outside options). That won't be entirely by having excellent people which will mean that internal research conversations are really good, but it seems to me like that was a significant part of what made FHI work (NB this wasn't just Nick, but people like Toby or Anders or Eric); I think it could be make-or-break for any new endeavour in a way that might be somewhat path-dependent in how it turns out; it seems right and proper to give it attention at this stage.


Makes sense! My inference was because the discussion at this stage is a high-level one about ways to set things up, but it does seem good to have space to discuss object-level projects that people might get into.


I agree in the abstract with the idea of looking for niches, and I think that several of these ideas have something to them. Nevertheless when I read the list of suggestions my overall feeling is that it's going in a slightly wrong direction, or missing the point, or something. I thought I'd have a go at articulating why, although I don't think I've got this to the point where I'd firmly stand behind it:

It seems to me like some of the central FHI virtues were:

  • Offering a space to top thinkers where the offer was pretty much "please come here and think about things that seem important in a collaborative truth-seeking environment"
    • I think that the freedom of direction, rather than focusing on an agenda or path to impact, was important for:
      • attracting talent
      • finding good underexplored ideas (b/c of course at the start of the thinking people don't know what's important)
    • Caveats:
      • This relies on your researchers having some good taste in what's important (so this needs to be part of what you select people on)
      • FHI also had some success launching research groups where people were hired to more focused things
        • I think this was not the heart of the FHI magic, though, but more like a particular type of entrepreneurship picking up and running with things from the core
  • Willingness to hang around at whiteboards for hours talking and exploring things that seemed interesting
    • With an attitude of "OK but can we just model this?" and diving straight into it
      • Someone once described FHI as "professional amateurs", which I think is apt
        • The approach is a bit like the attitude ascribed to physicists in this xkcd, but applied more to problems-that-nobody-has-good-answers-for than things-with-lots-of-existing-study (and with more willingness to dive into understanding existing fields when they're importantly relevant for the problem at hand)
    • Importantly mostly without directly asking "ok but where is this going? what can we do about it?"
      • Prioritization at a local level is somewhat ruthless, but is focused on "how do we better understand important dynamics?" and not "what has external impact in the world?"
  • Sometimes orienting to "which of our ideas does the world need to know about? what are the best ways to disseminate these?" and writing about those in high-quality ways
    • I'd draw some contrast with MIRI here, who I think were also good at getting people to think of interesting things, but less good at finding articulations that translated to broadly-accessible ideas

Reading your list, a bunch of it seems to be about decisions about what to work on or what locally to pursue. My feeling is that those are the types of questions which are largely best left open to future researchers to figure out, and that the appropriate focus right now is more like trying to work out how to create the environment which can lead to some of this stuff.

Overall, the take in the previous paragraph is slightly too strong. I think it is in fact good to think through these things to get a feeling for possible future directions. And I also think that some of the good paths towards building a group like this start out by picking a topic or two to convene people on and get them thinking about. But if places want to pick up the torch, I think it's really important to attend to the ways in which it was special that aren't necessarily well-represented in the current x-risk ecosystem.


I think FHI was an extremely special place and I was privileged to get to spend time there. 

I applaud attempts to continue its legacy. However, I'd feel gut-level more optimistic about plans that feel more grounded in thinking about how circumstances are different now, and then attempting to create the thing that is live and good given that, relative to attempting to copy FHI as closely as possible. 

Differences in circumstance

You mention not getting to lean on Bostrom's research taste as one driver of differences, and I think this is correct but may be worth tracing out the implications of even at the stage of early planning. Other things that seem salient and important to me:

  • For years, FHI was one of the only places in the world that you could seriously discuss many of these topics
    • There are now much bigger communities and other institutions where these topics are at least culturally permissible (and some of them, e.g. AI safety, are the subject of very active work)
    • This means that:
      • One of FHI's purposes was serving a crucial niche which is now less undersupplied
      • FHI benefited from being the obvious Schelling location to go to think about these topics
        • Whereas even in Berkeley you want to think a bit about how you sit in the ecosystem relative to Constellation (which I think has some important FHI-like virtues, although makes different tradeoffs and misses on others)
  • FHI benefited from the respectability of being part of the University
    • In terms of getting outsiders to take it seriously, getting meetings with interesting people, etc.
    • I'm not saying this was crucial for its success, and in any case the world looks different now; but I think it had some real impact and is worth bearing in mind
  • As you mention -- you have a campus!
    • I think it would be strange if this didn't have some impact on the shape of plans that would be optimal for you

Pathways to greatness

If I had to guess about the shape of plans that I think you might engage in that would lead to something deserving of the name "FHI of the West", they're less like "poll LessWrong for interest to discover if there's critical mass" (because I think that whether there's critical mass depends a lot on people's perceptions of what's there already, and because many of the people you might most want probably don't regularly read LessWrong), and more like thinking about pathways to scale gracefully while building momentum and support.

When I think about this, two ideas that seem to me like they'd make the plan more promising (that you could adopt separately or in conjunction) are (1) starting by finding research leads, and/or (2) starting small as-a-proportion-of-time. I'll elaborate on these:

Finding research leads

I think that Bostrom's taste was extremely important for FHI. There are a couple of levels this was true on:

  • Cutting through unimportant stuff in seminars
    • I think it's very easy for people, in research, to get fixated on things that don't really matter. Sometimes this is just about not asking enough which the really important questions are (or not being good enough at answering that); sometimes it's kind of performative, about people trying to show off how cool their work is
    • Nick had low tolerance for this, as well as excellent taste. He wasn't afraid to be a bit disagreeable in trying to get to the heart of things
    • This had a number of benefits:
      • Helping discussions in seminars to be well-focused
      • Teaching people (by example) how to do the cut-through-the-crap move
      • Shaping incentives for researchers in the institute, towards tackling the important questions head on
  • Gatekeeping access to the space
    • Bostrom was good at selecting people who would really contribute in this environment
      • This wasn't always the people who were keenest to be there; and saying "no" to people who would add a little bit but not enough (and dilute things) was probably quite important
      • In some cases this meant finding outsiders (e.g. professors elsewhere) to visit, and keeping things intellectually vibrant by having discussions with people with a wide range of current interests and expertise, rather than have FHI just become an echo chamber
  • Being a beacon
    • Nick had a lot of good ideas, which meant that people were interested to come and talk to him, or give seminars, etc.

If you want something to really thrive, at some point you're going to have to wrestle with who is providing these functions. I think that one thing you could do is to start with this piece. Rather than think about "who are all the people who might be part of this? does that sound like critical mass?", start by asking "who are the people who could be providing these core functions?". I'd guess if you brainstorm names you'll end up with like 10-30 that might be viable (if they were interested). Then I'd think about trying to approach them to see if you can persuade one or more to play this role. (For one thing, I think this could easily end up with people saying "yes" who wouldn't express interest on the current post, and that could help you in forming a strong nucleus.)

I say "leads" rather than "lead" because it seems to me decently likely that you're best aiming to have these responsibilities be shared over a small fellowship. (I'm not confident in this.)

Your answer might also be "I, Oliver, will play this role". My gut take would be excited for you to be like one of three people in this role (with strong co-leads, who are maybe complementary in the sense that they're strong at some styles of thinking you don't know exactly how to replicate), and kind of weakly pessimistic about you doing it alone. (It certainly might be that that pessimism is misplaced.)

Starting small as-a-proportion-of-time

Generally, things start a bit small, and then scale up. People can be reluctant to make a large change in life circumstance (like moving job or even city) for something where it's unknown what the thing they're joining even is. By starting small you get to iron out kinks and then move on from there.

Given that you have the campus, I'd seriously consider starting small not as-a-number-of-people but as-a-proportion-of-time. You might not have the convening power to get a bunch of great people to make this their full time job right now (especially if they don't have a good sense who their colleagues will be etc.). But you probably do have the convening power to get a bunch of great people to show up for a week or two, talk through big issues, and spark collaborations. 

I think that you could run some events like this. Maybe to start they're just kind of like conferences / workshops, with a certain focus. (I'd still start by trying to find something like "research leads" for the specific events, as I think it would help convening power as well as helping the things to go well.) In some sense that might be enough for carrying forward the spirit of FHI -- it's important that there are spaces for it, not that these spaces are open 365. But if it goes well and they seem productive, it could be expanded. Rather than just "research weeks", offer "visiting fellowships" where people take a (well-paid) 1-3 month sabbatical from their regular job to come and be in person all at the same time. And then if that's going well consider expanding to a permanent research group. (Or not! Perhaps the ephemeral nature of short-term things, and constantly having new people, would prove even more productive.)


It's a fair point that wisdom might not be straightforwardly safety-increasing. If someone wanted to explore e.g. assumptions/circumstances under which it is vs isn't, that would certainly be within scope for the competition.


Multiple entries are very welcome!

[With some kind of anti-munchkin caveat. Submitting your analyses of several different disjoint questions seems great; submitting two versions of largely the same basic content in different styles not so great. I'm not sure exactly how we'd handle it if someone did the latter, but we'd aim for something sensible that didn't incentivise people to have been silly about it.]

Load More