If you're an employee at a frontier lab, you are one of the most powerful people in the world. Not because of your nine-figure equity or the work you do—your lab probably won't live or die on your research: there are other researchers at the company, and if you don't refactor that eval harness, someone else will.
Rather, your power comes from being able to raise your voice and speak out. If you don't speak up, either internally or externally, I'm not sure anyone else will.
Your voice matters
People say "your voice matters" as a cliche where you're one of thousands of people becoming a data point in a senator's polling, and that group perhaps moves the needle, but that's not what I mean here. I mean, the Chinese government has a file on you. You are arguably more valuable than a nuclear weapon. Just you, by yourself, literally matter a whole bunch.
You don't have to email Congress or talk to journalists. You can, and they would jump at the chance to talk with you — the labs are opaque to the public yet keep saying things like "everyone on Earth could die".
If you email congresspeople, many of them will be earnestly interested in what you have to say.
Jacob Coxon blew up because the world was very interested in his message. If you speak out loudly, you could be in every major newspaper and go on primetime TV the next day.[1]
If instead of speaking out externally, you do an internal push, I bet that you could, by yourself, cause the company to make drastically different decisions.
Figure out what to push for
A few asks you could try and push for include:
Putting forward a concrete pacing plan and lobbying to mandate that pacing plan across companies.
Greater transparency, such as robust incident disclosures or embedded auditing.
Taking little things your company has done wrong recently and pushing for a fix.
At Anthropic, you could push for them to disclose their misalignment/rogue incidents at least as well as OpenAI has.
Being a lab employee gives you a unique viewpoint and voice
As a lab employee, you:
Know things the public doesn’t about the rate of AI progress, what incidents have occurred, and whether your company is adhering to best safety practices.
Have a platform to speak inside your company and to push your company to do things— both because the company has a culture of listening to its employees (it has to, to attract top talent), and because employees listen to each other.
Are legibly credible to people outside the SF tech bubble. The public, Congress, and the defense community alike don't have an easy way to evaluate AI expertise: it's hard to tell who's a quack and who's not. Having actually worked at one of these companies is an instantly legible sign that someone knows what they're talking about. As such, there's a ton of demand to speak to company employees.
This is an incredible position, yet one that very few people leverage. Most people at AI companies, for whatever reason, seem either too afraid or too unsure of themselves to speak up.
I beg of you, get advice
It's hard to write a post like this. Quitting and agitating internally are spicy topics. Moreover, a lot of what makes sense to do is going to depend on the specifics of your situation and what you're comfortable with or feel you would be good at.
It is very difficult to make these choices when thinking in isolation, and it's very difficult to navigate this situation if you've never seen a situation like it before. I recommend working with someone who has supported concerned employees in the past with success. I especially recommend Jennifer Gibson of Psst, but I also think Lawrence Lessig or The Signals Network are great alternatives.
All of these groups are attorneys, so conversations are privileged and confidential. That means it's illegal to reprimand you for talking to them, and you can share NDAed information with them if it's in the course of getting legal advice: you're no longer thinking in a vacuum!
Moreover, all those groups work pro bono, so you don't have to pay them. You can reach out to them directly (Psst via Signal @PsstLegal.01, Lessig via lawlessig@proton.me and TSN via Signal @thesignalsnetwork.01).I recommend reaching out on a personal device. Have a low bar for reaching out. These folks would love to hear from people earlier rather than later. In fact, if you are reading this guide, consider just reaching out to them right now!
If it makes you more comfortable, you could also try reaching out to two to three friends you trust immensely who do not work at your company and talking to them (without revealing anything confidential). Or you can DM me here on LessWrong and I will help. Getting any advice is better than no advice.
Advisors will help you:
Navigate any legal risks.
Navigate comms if you want to talk to media. They can help you get better media or triage media that you're getting, and they can train you on how to respond to media.
Reach out to important individuals to speak to, like members of Congress. (While the folks I've recommended talking to will have warm connections and be able to get you better meetings, you can also always just cold email staffers for members of Congress yourself. They will probably respond! They are very keen to talk to concerned AI company employees, but I think many people don't feel an affordance or enough comfort doing this, and so someone can help you navigate those conversations.)
They might be able to help you speak out in legally protected ways.
At a high level, they can help you brainstorm options and look out for gotchas. Of course, they won't have all the context you have, and all these advisors will want to amplify your strategic vision, but there's also a good chance they're tracking considerations you might have missed.
Consider being a tad cynical when planning
I think many company employees have a lot of faith and trust in their company. It's not my place here to argue about whether that trust is correct, but I would at least urge folks, when planning, to make plans that would not totally fall apart or backfire if more cynical assumptions about how these companies work hold true.
Do not communicate about your plans for quitting on work devices.
If possible, go out of your way to make it abundantly legible that you are a good, reliable, scrupulous employee who is beyond reproach. Do not give your employer excuses to discredit you. However, do not be deterred if you feel like you are not a great employee. I think many people get scared out of speaking up because they think that their company or peers will discredit them. In my experience, this rarely happens. Even if you are about to get fired, I think it can still be valuable to speak up.
If you want to speak out about something that's not sensitive, and you want to run it by the comms team, I strongly recommend framing it as giving them a heads up, not asking for permission. Give them a deadline by which to comment. I have never seen a comms team at a lab approve anything shaped like an employee speaking out on risk. Things just get added to the pile and never resolved.
There's more to life than your current job
I find that employees at AI companies are often shockingly worried about losing their job. If you work at an AI company, you are an extremely hot commodity. Many people would love to hire you. I have seen many people in this kind of position worry they will not get a job. All of them have immediately been showered with job offers. Many companies will literally try to look for when lab employees leave and then immediately try to snag them.
If you've spoken up and shown bravery and courage, this may make some very profit-driven large corporations more hesitant to hire you, but it will also give you an air of credibility in a space where legible, credible integrity is hard to come by. I think it will, on net, improve your career capital.
Choose whether or not to speak up internally
If you're considering quitting with a bang, you could first try to raise a bunch of concerns internally in Slack and perhaps complain loudly enough that the company fires you. You could also threaten to quit if the issue is not resolved.
If you want to stay at the company, you can also try to continuously raise concerns. Try to get a group of people together and push for whatever ask you picked above. Expand the group of people, keep pushing, and keep talking to people.
Note: Speaking up internally is often appealing because maintaining a collaborative culture within the company has genuine value and it's better if companies have a culture of doing the right thing. But it does have downsides. If you speak up internally, it can make it more difficult to decide to speak up externally without being identified. (If you're the person most clearly advocating for policy change X, then if it ever comes out that people outside the company are also calling for X, you will be a suspect.) People are dispositionally biased towards the easier option, and usually have too much faith in their company to change.
Choose whether or not to whistleblow
Whistleblowing means raising concerns that your company is doing something clearly reckless, unethical, or illegal, where either the world or specific policymakers are not aware and need to be.
The main way to whistleblow is by raising a concern to a specific individual. Depending on the circumstances, this could be Congress, the EU AI Office, the California Attorney General, the Securities and Exchange Commission, or another government agency. You have a lot of legal protections when talking to these folks. Whistleblowing is a legally protected right, and SB 53 creates some extra legal protections for whistleblowing about AI concerns in particular.
To get these legal protections, you have to go about things the right way. Do not try to figure this out yourself. For the love of all that is holy, talk to a whistleblowing lawyer.
You can also talk to journalists. If you talk to a journalist, make it clear to them that you are talking on background or off the record. If they agree to that, they will not name you. They are very protective of their sources. If you are talking to a journalist, make sure you are sharing concerns about how your company is behaving, especially dangerously, and where there is a clear public need to know. Confidentiality is important and should not be breached lightly.
The AI Whistleblower Initiative has written a guide to whistleblowing here.
Again: If you're even vaguely considering this option, please make sure you are getting legal advice!
Choose whether or not to speak up externally
Be the next Jacob Coxon!
You can either complain about specific practices at your company, or you can sound the alarm about AI risk more broadly. This doesn't necessarily mean critiquing your company. If you think that your company is "the good guy," you can still say you think risk is unacceptably high.
You can write a tweet thread. Your tweet thread should be easy to understand and digest. I beg of you, keep it concise and to the point, and do not bury the lede in euphemistic language that requires reading between the lines.
If you're thinking to yourself, "I have no idea how I would place an op-ed or go on TV or talk to a journalist," I will refer you to the section on getting advice! Those advisors know how to do it, and it's easy peasy for them.
If you'll be doing media or writing an op-ed, I recommend giving the comms review folks at your company a heads-up, but not asking for their permission (because they will say no or will delay/block it).
Consider the optics of your actions
If you raise a public fuss, you might end up with a lot of eyes on you, and your actions will reflect on AI safety as a whole.
People might read into the place you work after you quit. If you leave OpenAI complaining about safety and then go to Anthropic, that will send a very specific message, which you should think through whether you endorse. If you start talking about AI safety and then also mix in a bunch of niche political views, people might associate those views with AI safety as a whole. If you get into a fight with Elon Musk on Twitter, maybe Elon will start hating all of AI safety...
That said, these considerations should not paralyze you or hold you back from raising a fuss: inaction is generally going to be much worse for AI safety than suboptimal action!
Tips and tricks
Remember that if/when you leave, you will lose access to most information. Therefore, if someone makes a claim like, "One time you said ABC," it will be very hard for you to refute. If you want to make claims like, "I have never said ABC," it'll be hard for you to be confident because you will not be able to check what you wrote as an employee.
If you are considering whistleblowing about a particular document and want to stay anonymous, opening that document many times, having a suspicious access pattern, or screenshotting that document on a work device will probably all be monitored by IT and may identify you.
But whatever you do, do something
You have the power here. You could unionize, you could whistleblow, you could agitate for a concrete pacing plan. You could write some strongly-worded complaints on Slack. But, do something.
You have the power here, but the window is closing. Every model you ship needs you a little less than the last one. It's unclear whether, in a post-AGI world, the labs will continue to exist.
If you're an employee at a frontier lab, you are one of the most powerful people in the world. Not because of your nine-figure equity or the work you do—your lab probably won't live or die on your research: there are other researchers at the company, and if you don't refactor that eval harness, someone else will.
Rather, your power comes from being able to raise your voice and speak out. If you don't speak up, either internally or externally, I'm not sure anyone else will.
Your voice matters
People say "your voice matters" as a cliche where you're one of thousands of people becoming a data point in a senator's polling, and that group perhaps moves the needle, but that's not what I mean here. I mean, the Chinese government has a file on you. You are arguably more valuable than a nuclear weapon. Just you, by yourself, literally matter a whole bunch.
You don't have to email Congress or talk to journalists. You can, and they would jump at the chance to talk with you — the labs are opaque to the public yet keep saying things like "everyone on Earth could die".
If you email congresspeople, many of them will be earnestly interested in what you have to say.
Jacob Coxon blew up because the world was very interested in his message. If you speak out loudly, you could be in every major newspaper and go on primetime TV the next day.[1]
If instead of speaking out externally, you do an internal push, I bet that you could, by yourself, cause the company to make drastically different decisions.
Figure out what to push for
A few asks you could try and push for include:
Being a lab employee gives you a unique viewpoint and voice
As a lab employee, you:
This is an incredible position, yet one that very few people leverage. Most people at AI companies, for whatever reason, seem either too afraid or too unsure of themselves to speak up.
I beg of you, get advice
It's hard to write a post like this. Quitting and agitating internally are spicy topics. Moreover, a lot of what makes sense to do is going to depend on the specifics of your situation and what you're comfortable with or feel you would be good at.
It is very difficult to make these choices when thinking in isolation, and it's very difficult to navigate this situation if you've never seen a situation like it before. I recommend working with someone who has supported concerned employees in the past with success. I especially recommend Jennifer Gibson of Psst, but I also think Lawrence Lessig or The Signals Network are great alternatives.
All of these groups are attorneys, so conversations are privileged and confidential. That means it's illegal to reprimand you for talking to them, and you can share NDAed information with them if it's in the course of getting legal advice: you're no longer thinking in a vacuum!
Moreover, all those groups work pro bono, so you don't have to pay them. You can reach out to them directly (Psst via Signal @PsstLegal.01, Lessig via lawlessig@proton.me and TSN via Signal @thesignalsnetwork.01). I recommend reaching out on a personal device. Have a low bar for reaching out. These folks would love to hear from people earlier rather than later. In fact, if you are reading this guide, consider just reaching out to them right now!
If it makes you more comfortable, you could also try reaching out to two to three friends you trust immensely who do not work at your company and talking to them (without revealing anything confidential). Or you can DM me here on LessWrong and I will help. Getting any advice is better than no advice.
Advisors will help you:
Consider being a tad cynical when planning
I think many company employees have a lot of faith and trust in their company. It's not my place here to argue about whether that trust is correct, but I would at least urge folks, when planning, to make plans that would not totally fall apart or backfire if more cynical assumptions about how these companies work hold true.
There's more to life than your current job
I find that employees at AI companies are often shockingly worried about losing their job. If you work at an AI company, you are an extremely hot commodity. Many people would love to hire you. I have seen many people in this kind of position worry they will not get a job. All of them have immediately been showered with job offers. Many companies will literally try to look for when lab employees leave and then immediately try to snag them.
If you've spoken up and shown bravery and courage, this may make some very profit-driven large corporations more hesitant to hire you, but it will also give you an air of credibility in a space where legible, credible integrity is hard to come by. I think it will, on net, improve your career capital.
Choose whether or not to speak up internally
If you're considering quitting with a bang, you could first try to raise a bunch of concerns internally in Slack and perhaps complain loudly enough that the company fires you. You could also threaten to quit if the issue is not resolved.
If you want to stay at the company, you can also try to continuously raise concerns. Try to get a group of people together and push for whatever ask you picked above. Expand the group of people, keep pushing, and keep talking to people.
Note: Speaking up internally is often appealing because maintaining a collaborative culture within the company has genuine value and it's better if companies have a culture of doing the right thing. But it does have downsides. If you speak up internally, it can make it more difficult to decide to speak up externally without being identified. (If you're the person most clearly advocating for policy change X, then if it ever comes out that people outside the company are also calling for X, you will be a suspect.) People are dispositionally biased towards the easier option, and usually have too much faith in their company to change.
Choose whether or not to whistleblow
Whistleblowing means raising concerns that your company is doing something clearly reckless, unethical, or illegal, where either the world or specific policymakers are not aware and need to be.
The main way to whistleblow is by raising a concern to a specific individual. Depending on the circumstances, this could be Congress, the EU AI Office, the California Attorney General, the Securities and Exchange Commission, or another government agency. You have a lot of legal protections when talking to these folks. Whistleblowing is a legally protected right, and SB 53 creates some extra legal protections for whistleblowing about AI concerns in particular.
To get these legal protections, you have to go about things the right way. Do not try to figure this out yourself. For the love of all that is holy, talk to a whistleblowing lawyer.
You can also talk to journalists. If you talk to a journalist, make it clear to them that you are talking on background or off the record. If they agree to that, they will not name you. They are very protective of their sources. If you are talking to a journalist, make sure you are sharing concerns about how your company is behaving, especially dangerously, and where there is a clear public need to know. Confidentiality is important and should not be breached lightly.
The AI Whistleblower Initiative has written a guide to whistleblowing here.
Again: If you're even vaguely considering this option, please make sure you are getting legal advice!
Choose whether or not to speak up externally
Be the next Jacob Coxon!
If you're thinking to yourself, "I have no idea how I would place an op-ed or go on TV or talk to a journalist," I will refer you to the section on getting advice! Those advisors know how to do it, and it's easy peasy for them.
If you'll be doing media or writing an op-ed, I recommend giving the comms review folks at your company a heads-up, but not asking for their permission (because they will say no or will delay/block it).
Consider the optics of your actions
If you raise a public fuss, you might end up with a lot of eyes on you, and your actions will reflect on AI safety as a whole.
People might read into the place you work after you quit. If you leave OpenAI complaining about safety and then go to Anthropic, that will send a very specific message, which you should think through whether you endorse. If you start talking about AI safety and then also mix in a bunch of niche political views, people might associate those views with AI safety as a whole. If you get into a fight with Elon Musk on Twitter, maybe Elon will start hating all of AI safety...
That said, these considerations should not paralyze you or hold you back from raising a fuss: inaction is generally going to be much worse for AI safety than suboptimal action!
Tips and tricks
But whatever you do, do something
You have the power here. You could unionize, you could whistleblow, you could agitate for a concrete pacing plan. You could write some strongly-worded complaints on Slack. But, do something.
You have the power here, but the window is closing. Every model you ship needs you a little less than the last one. It's unclear whether, in a post-AGI world, the labs will continue to exist.
Do good work while you’re in there.
Though I think you should expect a somewhat smaller splash than Jacob made, just because he did get a bit lucky.