Quick Takes

I just completed the Horizon Institute's AI policy workshop in DC, a three-day event with the explicit goal of placing safety and technically-minded people in policy roles. As I understand it, they do very little vetting of the particular beliefs about AI that participants hold, trusting instead that putting smart people in good positions will bring about good outcomes. The mission is awesome, and they are great at what they do. I highly recommend applying for their workshops and fellowships if you are AI policy-curious. Here are a few of my general takeaways from the event. I omit specific names due to the closed door nature of the event, which benefits from the candor of its guests and participants alike. * Policy people are concerned. Half the guest speakers currently doing AI policy work were X-risk level scared and took agentic risks seriously. This view was even more common among us participants—at least 80% of participants by my estimation. * Well positioned people can make a difference. AGI-pilled congressional staffers have had some success in persuading their AI-curious legislators to take X-risk seriously. Such staffers can play an outsized role in what gets endorsed and what bills advance. Think Tank work can make these issues more salient to legislators. Executive branch workers can produce policy proposals that get put into place. One can, in fact, influence events. * An insider estimated that about 12 congressional staffers are AGI and X-risk-pilled in the House of Representatives, without clear party lines. The glass is not half full yet, but 12/435 full and blessedly purple? Many more staffers are worried about AI in other ways (education/data centers/deep fakes). * Networking is the name of the game in DC. Apparently it is the only way to get a job at the Think Tanks and in Congressional staff, and networking remains 50% of the job after hiring. Credentials matter a tiny bit in the Senate and much less in the House, though there are committe

LessWrong’s had a lot of discourse over the years about the virtue of brevity. However, that could hypothetically mean a few things. * People want others to do a better job editing their posts. * People want writers to assume more shared context, or a greater capacity of readers to independently infer what’s meant, so that more can be left implicit. * When there’s an intrinsic tradeoff between nuance and brevity, people prefer more brevity on current margins. * People don’t actually care about brevity and are being sarcastic. It’s also possible the situation has changed and the notion that there’s a desire for more brevity is just out of date. I’m not sure if any of these are true, but my tentative impression is that LessWrong’s karma voting patterns reflect a combination of factors: 1. LessWrong sincerely wants more brevity, but isn’t always explicit or clear about the tradeoffs in the writing content that would require. 2. LessWrong’s desire for brevity shouldn’t be conflated with a desire for less discourse or a restriction of discourse to what’s intellectually novel. In this, LessWrong is perhaps less similar to academia and more similar to a think tank. LessWrong rewards making arguments that may be perceived as capable of broadcasting its shared views to a wider readership, or as defining what the existing community stands for. To oversimplify, LessWrong appreciates it when writers spend words to say out loud what everyone around here already thinks. 3. LessWrong dislikes hyperbole, and rewards explicit, sometimes lengthy arguments against hyperbolic statements, as well as explicit flagging and caveats. This is true even when everyone knows it’s hyperbole (see what I did there?). Hyperbole is common and rewarded outside of LessWrong, and since hyperbolic statements are often also short and to the point, it can be easy to conflate with the kind of nuance-preserving brevity LessWrong actually rewards. 4. LessWrong rewards brevity achieved via better e

[Fiction] Medjool the Watcher stands at the Gate of the World. The centuries crowd there, eager for admittance. But Medjool decides. In Aegypt, Medjool let a handful of centuries play for three thousand years, then let newer centuries sweep them away. In the Southern Continent, Medjool allowed a single century forty millennia, then banished it to the void. The centuries come so fast now, that each brings some new wonder into the world. Some say that Medjool no longer stands watch, and the centuries press through the Gate pell-mell. Surely the Fûm is upon us, the chaotic end of all things.

Specification gaming is way worse than I expected (cf HF hacks), and I think reward might empirically become a primary optimization target (not just a secondary priority for AI agents, as I had hypothesized in 2022). * I found the Apollo paper quite convincing on the reward point. The reward-seeking doesn't look like it's due to self-fulfilling misalignment (since before RL the systems weren't retargetable according to their beliefs about the reward model). * My main mistake in 2022 was not appreciating how LLM pretraining would affect the concepts available to an AI. Namely, by the time RL started, the systems would already know about the "reward" concept. In 2022, I had flagged "how does pretraining affect this reasoning" as a known unknown. (Flagging a known unknown doesn't stop it from blowing up your reasoning!) * Oops Reward is still not definitionally the optimization target. The equations themselves still don't tell you "yes this will train something that seeks its reinforcement signal." (That'd prove too much, as some people don't seek their reinforcement signals despite knowing about them, although their "equations" are not the same as the equations of an AI's training process.) This theoretical point is the other half of my 2022 piece and I stand by it. Saying "systems are trained to get reward" is still a mistake and degrades precision of thought because "reward" has overly delicious connotations. "Reinforcement" is usually a better term. Whether or not smart AI systems empirically seek to make the reinforcement signal high, we need to think clearly and evenly about the conditions that push towards / away from that outcome. (So I would instead say: "Reward is not definitionally the optimization target.") If the HF swarm-agents had been reinforcement-seeking, they would have tried to hack their reinforcement processes. AFAIK that didn't happen and they tried to complete the task (in a twisted way). Seems important. I have a post drafted out about

According to many thinkpieces: people are reading less, students are cheating more, professors are angry, English professors have to treat their class like it's remedial, high school teachers are crying students can't write a full paragraph. According to the chess community: everyone is better, across all levels. GMs train against computers, beginners have multimedia learning resources and can have their games auto-critiqued, intermediates get exposed to chess opening theory sooner, and everyone is playing more games with all sorts of people. The "chess" experience is pretty techno-optimist, and since techno-pessimism is in vogue, you have to almost pay attention more carefully to notice the pocket of techno-optimism. Has anyone tried squaring these two viewpoints? They seem pretty directionally opposed.

How would we get to a "social victory" regarding AGI X-risk? There's the general goal of slowing down or stopping progress toward AGI, e.g. through banning large research investments. There are lots of potential approaches; see Grace's 2022 Let’s think about slowing down AI and this list by Zach Stein-Perlman. In the medium-term, it's quite hard to prevent progress from continuing. Progress can come from theoretical work. It can also come from smaller computations (e.g. small training runs) that are easier to do in secret and that require fewer resources. (See Red vs. Blue research.) In the medium-term, how do we avoid continuing to make progress toward AGI, given that? One type of approach is to demotivate individuals from doing that sort of research. Some subtypes: * convince AI researchers that AGI is a bad idea * provide alternative fun / interesting / cool / lucrative things for AI researchers to do instead * heal the souls of AI researchers and/or their surrounding culture, so that they can stop being so nihilistic or selfish or deluded or similar * provide alternative hopes for achieving the goods that would hypothetically come from AGI, e.g. via human intelligence amplification * use social pressure * use legal restriction * use professional norms, e.g. you're fired from the university if you're doing AGI research * use formal signaling (that is, use legal restriction, professional norms, etc., as a way to signal "society really doesn't like this") My attention is drawn to the idea of using social pressure, because that one seems to be on a pareto frontier of effectiveness. In particular, social pressure can be pretty pervasive and effective even against things that aren't that hard to do covertly, though not perfectly effective. In contrast, many other methods are either not pervasive (e.g. you can convince many people to stop because it's bad on their inside view, but probably not 99.99% of people) or not effective against some covert behavior

i wish more people were aware that safety research epistemics at labs are extremely distorted, and that this is a downside worth considering when deciding where to do safety research or how one ought to comport oneself if choosing to work on safety at a lab