Irresponsible Companies Can Be Made of Responsible Employees
88leogao
8samuelshadrach
13leogao
4Yeonwoo Kim
2Bronson Schoen
1samuelshadrach
1VojtaKovarik
4VojtaKovarik
4leogao
1Bronson Schoen
2habryka
6Fabien Roger
2Soapspud
2VojtaKovarik
1Soapspud
1VojtaKovarik
New Comment
my guess:
- selective hiring is very real. lots of people who are xrisk pilled just refuse to join oai. people who care a lot often end up very stressed and leave in large part because of the stress.
- the vast majority of people at oai do not think of xrisk from agi as a serious thing. but then again probably a majority dont really truly think of agi as a serious thing.
- people absolutely do argue "well if i didn't do it, someone else would. and even if oai stopped, some other company would do it" to justify their work.
- compartmentalization is probably not a big part of the reason, at least not yet. historically things don't get compartmentalized often, and even when they do, i don't think it makes the difference between being worried and not being worried about xrisk for that many people
- as companies get big, teams A B C not talking to each other is the default order of the world and it takes increasing effort to get them to talk to each other. and even getting them talking is not enough to change their courses of action, which often requires a lot of work from higher up. this hampers everything; this is in general why big companies have so many overlapping/redundant teams
- people get promoted / allocated more resources if they do things that are obviously useful for the company, as opposed to less obviously useful for the company (i mean, as a company, you kind of understandably have to do this or else die of resource misallocation).
- i think quite a few people, especially more senior people, are no longer driven by financial gain. these things are sometimes "i really want to accomplish something great in the field of ML" or "i like writing code" or "i like being part of something important / shaping the future". my guess is anyone super competent who cares primarily about money quits after a few years and, depending on the concavity of their utility function, either retires on a beach, or founds a startup and raises a gazillion dollars from VCs
- it's pretty difficult to do weird abstract bullshit that doesn't obviously tie into some kind of real world use case (or fit into the internally-accepted research roadmap to AGI). this has imo hampered both alignment and capabilities. it makes a lot of sense though, like, bell labs didn't capture most of the value that bell labs created, and academia is the place where weird abstract bullshit is supposed to live, and we're in some sense quite lucky that industry is willing to fund any of it at all
- concretely this means anything alignmenty gets a huge boost if you can argue that it will (a) improve capabilities or (b) prevent some kind of embarrassing safety failure in the model we're currently serving to gazillions of people. the kinds of things people choose to work on are strongly shaped by this as a result, and probably explains why so much work keeps taking alignment words and using them to mean aligning GPT-5 rather than AGI.
- aside from the leopold situation, which had pretty complicated circumstances, people don't really get fired for caring about xrisk. the few incidents are hard to interpret because of strong confounding factors and could be argued either way. but it's not so far from base rates so i don't feel like it's a huge thing.
- my guess is a lot of antipathy towards safety comes from broader antipathy against safetyism as a whole in society, which honestly i (and many people in alignment) have to admit some sympathy towards.
3
1
1I am not surprised to hear this but also, this is insane.
All the lab heads are repeatedly publicly claiming they could cause human extinction, superintelligence is within reach, and a majority of people at their own labs don't take them seriously on this.
I'm somewhat confused what causes a group of people who talk to each other everyday, work on the same projects, observe the same evidence, etc to come to such wildly different conclusions about the work they're doing together and then be uninterested in resolving the disagreement.
Is there a taboo being enforced against discussing these disagreements inside the labs?
this is pretty normal? it's really hard for leadership to make employees care about or believe specific things. do you really think the average Amazon employee or whatever has strong opinions on the future of delivery drones? does the average Waymo employee have extremely strong beliefs about the future of self driving?
for most people in the world, their job is just a job. people obviously avoid working on things they believe are completely doomed, and tend to work on cool trendy things. but generally most people do not really have strong beliefs about where the stuff they're working on is going.
no specific taboo is required to ensure that people don't really iron out deep philosophical disagreements with their coworkers. people care about all sorts of other things in life. they care about money, they care whether they're enjoying the work, they care whether their coworkers are pleasant to be around, they care about their wife and kids and house.
once you have a company with more than 10 people, it requires constant effort to maintain culture. hiring is way harder if you can only hire people who are aligned, or if you insist on aligning people. if you grow very fast (and openai has grown very fast - it's approximately doubled every single year I've been here), it's inevitable that your culture will splinter. forget about having everyone on the same page; you're going to have entire little googletowns and amazontowns and so on of people who bring Google or Amazon culture with them and agglomerate with other recent transplants from those companies.
I agree with your point. I think this tendency is especially visible in global conglomerates with rigid personnel structures. Take Samsung, for example — a company that supplies memory semiconductors while working with Nvidia and TSMC in the AI space. Samsung’s influence is comparable to that of many Silicon Valley giants, but its HR system is still based on traditional corporate aptitude tests and academic pedigree.
They hire on a massive scale, and the problem isn’t really about whether someone believes in X-risk or not — sometimes promotions are limited or even employment is terminated simply because of age. In more flexible job markets, like the Bay Area, the issue you described seems much less pronounced.
I’m writing this from the perspective of Seoul, where the job market is very isolated and rigid, so my comment doesn’t represent the whole world. I just wanted to say that, in that sense, what you wrote might actually make more sense on a global level. On the other side of the planet, there are people who worry less about whether the AI they’re building might end the world, and more about how they’ll feed their families tomorrow.
VojtaKovarik expressed that beautifully through the concept of fiducity, but from an individual’s point of view, it can be summed up more simply as: “One meal tomorrow matters more than the end of the world ten years from now.”
I think it’s notably abnormal specifically because it wasn’t the “default” equilibrium for OpenAI specifically.
Like earlier you mentioned:
selective hiring is very real. lots of people who are xrisk pilled just refuse to join oai. people who care a lot often end up very stressed and leave in large part because of the stress.
and
most cases of xrisk people leaving are just because people felt sidelined/unhappy and chose to leave.
One model of this is “its normal that people at any company don’t have strong opinions about their work”. Another model is “lots of people in various positions did in fact have strong opinions about this given the stakes and left”.
If you send such strong signals about safety that people preemptively filter out of the hiring pipeline, then people who are already there with strong opinions on safety feel sidelined, IMO the obvious interpretation is “you actively filtered against people with strong views on safety”.
I appreciate your answer, it adds useful info.
do you really think the average Amazon employee or whatever has strong opinions on the future of delivery drones? does the average Waymo employee have extremely strong beliefs about the future of self driving?
I assume yes a lot of employees do believe technical capabilities are further ahead of what the rest of the public is aware of, because they have insider knowledge. And they are aware they are accelerating it.
They might not deeply care about drones but they are aware what is going on.
With AI, you can't not care about ASI or extinction or catastrophe or dictatorship or any of the words thrown around, as they directly affect your life too.
once you have a company with more than 10 people, it requires constant effort to maintain culture. hiring is way harder if you can only hire people who are aligned, or if you insist on aligning people.
Then I expect OpenAI to splinter hard as we get closer to ASI.
With AI, you can't not care about ASI or extinction or catastrophe or dictatorship or any of the words thrown around, as they directly affect your life too.
I think it is important to distinguish between caring in the sense of "it hugely affects your life" and caring in the sense of, for example, "having opinions on it, voicing them, or even taking costlier actions". Presumably, if something affects you hugely, you should care about it in the second sense, but that doesn't mean people do. My uninformed intuition is that this would apply to most employees of AI companies.
Can you gesture at what you are basing this guess on?
Ignore this if you are busy, but I was also wondering: Do you have any takes on "do people get fired over acting on beliefs about xrisk", as opposed to "people getting fired over caring about xrisk"?
And perhaps on "whether people would get fired over acting on xrisk beliefs, so they don't act on them"? (Though this seems difficult to operationalise.)
as far as I'm aware, the only person who can be argued to have ever been fired for acting on beliefs about x risk is leopold, and the circumstances there are pretty complicated. since I don't think he's the only person to have ever acted on xrisk at oai to the extent he did, I don't think this is just because other people don't do anything about xrisk.
most cases of xrisk people leaving are just because people felt sidelined/unhappy and chose to leave. which is ofc also bad, but quite different.
I don't think this is just because other people don't do anything about xrisk.
Why is that? If someone went off and consistently worked on an agenda that was directly xrisk related (that didn’t contribute to a short term capabilities or product safety) you’re saying they wouldn’t get sidelined / not allocated resources / fired?
Another dynamic is that employees at AI companies often think that their AI company not going bankrupt will allow P(doom) reduction, at least via the classic mechanism of "having more power lets us do more good things" (e.g. advocating for good policies using the clout of a leading AI company, doing the cheap and important safety/security stuff that the counterfactual company would not even bother doing - and demonstrating that these cheap and important things are cheap and important, using AIs in net-good ways that other companies would have not bothered doing, not trying to spread dangerous ideologies, ...).
And it's unclear how incorrect this reasoning is. This seems to me like a sensible thing to do if you know the company will actually use its power for good - but it's worrisome that so many people in so many different AI companies think they are in an AI company that will do something better with the power than the company that would replace them if they went bankrupt.
I think that "the company is doing sth first-order risky but employees think it's actually good because going bankrupt would shift power towards some other more irresponsible actors" is at least as central as "the company is doing something risky to avoid going bankrupt and employees would prefer that the company would not do it, but [dynamic] prevents employees from stopping it".
But I would be curious to know what are the dynamics that actually take place, which of them matter how much, and what is the overall effect.
I would be interested to know more about those too! However, I don't have any direct experience with the insides of AI companies, and I don't have any friends who do either, so I'm hoping that other readers of this post might have insights that they are willing to share.
For those who have worked for AI companies or have reliable info from others who have worked for an AI company, these are a few things I am especially curious about, categorised by the mechanisms mentioned in the post:
Selective hiring
- What kinds of personality traits are application-reviewers and interviewers paying attention for? To what extent are the following traits advantages or disadvantages to candidates getting hired?
- bias to action?
- agreeableness?
- outspokenness?
- conscientiousness?
- Ignoring any explicit criteria that are supposed to factor into the decision to hire someone, what was the typical distribution of personality traits in people who were actually hired?
- What about the distribution of traits in people who continued to work there for longer than two years?
- To what extent are potential hires expected to have thought about the wider impacts of the work they will be doing?
- Does an applicant's level of concern on catastrophic risk typically come up during the hiring process? Is that a thing that factors into the decision to hire them? (Ignoring the question of whether it should be a factor for now.)
Firing dissenters as a coordination problem
- How often have employees had consequential disagreements with the team or company's direction and voiced it?
- How often have employees had consequential disagreements with the team or company's direction and not voiced it? Why not?
- How often do employees in the company see their colleagues express consequential disagreements with the work they are doing?
- How often have you seen employees express consequential disagreements and then have them addressed to their satisfaction?
- Among the employees who have expressed consequential disagreements at some point, how many of them work on the same team two years later?
- How many of them still work at the company at all two years later?
Compartmentalising undesirable information (and other types of information control)
- How long has the average Risk Team member worked for that team?
- How often do people on the teams addressing risks talk to people outside their teams about the implications of their work?
- How often do people working on the teams addressing risks find themselves softening or otherwise watering down their risk assessments/recommendations when communicating with people outside their teams?
Yep, these all seem relevant, and I would be interested in answers. (And my thanks to leogao for their takes.)
I would additionally highlight[1] the complication where even if a tendency is present -- say, selective hiring for a particular belief -- it might not work as explicitly as people knowingly paying attention to it. It could be implicitly present in something else ("culture fit"), or purely correlational, etc. I am not sure how to best deal with this.
FWIW, I think this concern is important, but we need to be very cautious about it. Since one could always go "yeah, your results say nothing like this is going on, but the real mechanism is even more indirect". I imagine that the fields like fairness & bias have to encounter this a lot, so they might be some insights.
- ^
To be clear, your comment seems aware of this issue. I just wanted to emphasise it.
it might not work as explicitly as people knowingly paying attention to it. It could be implicitly present in something else ("culture fit")
Yes, that sounds plausible to me as well. I did not mention those because I found it much harder to think of ways to tell when those dynamics are actually in play.
If I understand this correctly:
FWIW, I think this concern is important, but we need to be very cautious about it. Since one could always go "yeah, your results say nothing like this is going on, but the real mechanism is even more indirect".
I think you are gesturing at the same issue?
I imagine that the fields like fairness & bias have to encounter this a lot, so they might be some insights.
It makes sense to me that the implicit pathways for these dynamics would be an area of interest to the fields of fairness and bias. But I would not expect them to have any better tools for identifying causes and mechanisms than anyone else[1]. What kinds of insights would you expect those fields to offer?
To be clear, I have only a superficial awareness of the fields of fairness and bias. ↩︎
> I imagine that the fields like fairness & bias have to encounter this a lot, so they might be some insights.
It makes sense to me that the implicit pathways for these dynamics would be an area of interest to the fields of fairness and bias. But I would not expect them to have any better tools for identifying causes and mechanisms than anyone else[1]. What kinds of insights would you expect those fields to offer?
[Rephrasing to require less context.] Consider questions "Are companies 'trying to' override the safety concerns of their employees?" and "Are companies 'trying to' hire fewer women or paying them less?". I imagine that both of these will suffer from the similar issues: (1) Even if a company is doing the bad thing, it might not be doing it through explicit means like having an explicit hiring policy. (2) At the same time, you can probably always postulate one more level of indirection, and end up going on witch hunt even in places where there are no witches.
Mostly, it just seemed to me that fairness & bias might be the biggest examples of where (1) and (2) are in tension. (Which might have something to do with there being both strong incentives to discriminate and strong taboos against it.) So it seems more likely that somebody there would have insights about how to fight it, compared to other fields like physics or AI, and perhaps even more than economics and politics. (Of course, "somebody having insights" is consistent with "most people are doing it wrong".)
As to how those insights would look like: I don't know :-( . Could just be a collection of clear historical examples where we definitely know that something bad was going on but naive methods X, Y, Z didn't show it, together with some opposite examples of where nothing was wrong but people kept looking until they found some signs? Plus some heuristics for how to distinguish these.
Curated and popular this week
