Poll For Topics of Discussion and Disagreement
Use this thread to (a) upvote topics you're interested in reading about, (b) agree/disagree with positions, and (c) add new positions for people to vote on.
Note: Hit cmd-f or ctrl-f (whatever normally opens search) to automatically expand all of the poll options below.
LLMs as currently trained run ~0 risk of catastrophic instrumental convergence even if scaled up with 1000x more compute
53
29
5
2Academia is sufficiently dysfunctional that if you want to make a great scientific discovery you should basically do it outside of academia.
583117Pursuing plans that cognitively enhance humans while delaying AGI should be our top strategy for avoiding AGI risk
63146Current progress in AI governance will translate with greater than 50% probability into more than a 2 year counterfactual delay of dangerous AI systems
351916It is very unlikely AI causes an existential catastrophe (Bostrom or Ord definition) but doesn't result in human extinction. (That is, non-extinction AI x-risk scenarios are unlikely)
373410Things will basically be fine regarding job loss and unemployment due to AI in the next several years and those worries are overstated
332714
2The current AI x-risk grantmaking ecosystem is bad and could be improved substantially.
32123People aged 12 to 18 should basically be treated like adults rather than basically treated like children.
47306
6
4It is critically important for US/EU companies to build AGI before Chinese companies.
452143EAs and rationalists should strongly consider having lots more children than they currently are
454210
2Meaningness's "Geeks Mops and Sociopaths" model is an accurate model of the dynamics that underlie most social movements
201912Someone in the AI safety community (e.g. Yud, Critch, Salamon, you) can currently, within 6 month's effort, write a 20,000 word document that would pass a threshold for a coordination takeoff on Earth, given that 1 million smart Americans and Europeans would read all of it and intended to try out much of the advice (i.e. the doc succeeds given 1m serious reads, it doesn't need to cause 1m serious reads). Copy-pasting already-written documents/posts would count.
461413
51
1It was a mistake to increase salaries in the broader EA/Rationality/AI-Alignment ecosystem between 2019 and 2022
321410Good AGI-notkilleveryoneism-conscious researchers should in general prioritize working at big AGI labs over working independently, for alignment-focused labs, or for academia marginally more than they currently do.
1055The ratio of good alignment work done at labs vs independently mostly skews toward labs
Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than independent researchers. Your answer should factor out such considerations.
Edit: Also normalize for quantity of researchers.
863There is a greater than 20% chance that the Effective Altruism movement has been net negative for the world.
502222A basic deontological and straightforward morality (such as that exmplified by Hermione in HPMOR) is basically right; this is in contrast with counterintuitive moralities that suggest evil-tinted people (like Quirrell in HPMOR) are also valid ways of being moral.
24225Just like the last 12 months was the time of the chatbots, the next 12 months will be the time of agent-like AI product releases.
261211The work of agency-adjacent research communities such as artificial life, complexity science and active inference is at least as relevant to AI alignment as LessWrong-style agent foundations research is.
865American intelligence agencies consider AI safety to be substantially more worth watching than most social movements
19129
8It is possible to make meaningful progress on deceptive alignment using experiments on current models
971Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large compute budgets for safety research teams.
2881"Polyamory-as-a-default-option" would be a better social standard than "Monogamy-as-a-default-option".
24133The rationality community will noticeably spill over into other parts of society in the next ten years. Examples: entertainment, politics, media, art, sports, education etc.
129At least one American intelligence agency is concerned about the AI safety movement potentially decelerating the American AI industry, against the administration/natsec community's wishes
151385I broadly agree with the claim that "most people don't do anything and the world is very boring".
24203
2
11
1On the current margin most people would be better off involving more text-based communication in their lives than in-person communication.
60851Agent foundations research should become more academic on the margin (for example by increasing the paper to blogpost ratio, and by putting more effort into relating new work to existing literature).
1072Current progress in AI governance will translate with greater than 20% probability into more than a 2 year counterfactual delay of dangerous AI systems
2154It is possible to make meaningful progress on ELK using empirical experiments on current models
87Language model agents are likely (>20%) to produce AGI (including the generalization to foundation model-based cognitive architectures)
1674Current AI safety university groups are overall a good idea and helpful, in expectation, for reducing AI existential risk
2774Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large-scale lobbying efforts in DC.
24121Immersion into phenomena is better for understanding them than trying to think through at the gears-level, on the margin for most people who read LessWrong.
271272Among existing alignment research agendas/projects, Superalignment has the highest expected value
1462Most LWers should rely less on norms of their own (or the LW community's) design, and instead defer to regular societal norms more.
3215"Open-source LLM-based agent with hacking abilities starts spreading itself over the Internet because some user asked it to do so or to do something like to conquer the world" is a quite probable point-of-no-return regarding AGI risk.
1431Investing in early-stage AGI companies helps with reducing x-risk (via mission hedging, having board seats, shareholder activism)
2342Rationality should be practiced for Rationality’s sake (rather than for the sake of x-risk).
431722
1The younger generation of rationalists are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.
831At least one of {Anthropic, OpenAI, Deepmind} is net-positive compared to the counterfactual where just before founding the company, its founders were all discretely paid $10B by a time-travelling PauseAI activist not to found the company and to exit the industry for 30 years, and this worked.
851The most valuable new people joining AI safety will usually take ~1-3 years of effort to begin to be adequately sorted and acknowledged for their worth, unless they are unusually good at self-promotion e.g. gift of gab, networking experience, and stellar resume.
1541There is a greater than 80% chance that effective altruism has been net-negative for the world.
4732If rationality took off in China, it would yield higher EV from potentially spreading to the rest of the world than from potentially accelerating China.
1384Effective altruism can be well modeled by cynically thinking of it as just another social movement, in the sense that those a part of it are mainly jockeying for in-group status, and making costly demonstrations to their in-group & friends that they care about other sentiences more than others in the in-group. Its just that EA has more cerebral standards than others.
27921Language model agents are very likely (>80%) to produce AGI (including the generalization to foundation model-based cognitive architectures)
1221Research into getting a mechanistic understanding of the brain for purposes of at least one of: understanding how values/empathy works in people, brain uploading or improving cryonics/plastination is net positive and currently greatly underfunded.
1231Most persistent disagreements can more usefully be thought of as a difference in priors rather than a difference in evidence or rationality.
286A Secular Solstice variation designed to work weekly (akin to Sunday Service or Shabbat) would be positive for rationalists, both for community and for the thought processes of the members.
24116
1There is a greater than 50% chance that the Effective Altruism movement has been net negative for the world.
4264I really like the idea of this page and gave this post a strong-upvote. Felt like this was worth mentioning, since in recent years I've felt increasingly alienated by LessWrong culture. My only major request here is that, if there are future iterations of this page, I'd like poll options to be solicited/submitted before any voting happens (this is so that early submissions don't get an unfair advantage just by having more eyeballs on them). A second more minor request is to hide the votes while I'm still voting (I'm trying very hard not to be influenced by vote counts and the names of specific people agreeing/disagreeing with things, but it's difficult).
I feel like LessWrong should just update-all-the-way and ask Manifold Markets for a stylable embed system.
1I work at Manifold, I don't know if this is true but I can easily generate some arguments against:
- Manifold's business model is shaky and Manifold may well not exist in 3 years.
- Manifold's codebase is also shaky and would not survive Manifold-the-company dying right now.
- Manifold is quite short on engineering labor.
- It seems to me that Manifold and LW have quite different values (Manifold has a typical startup focus on prioritizing growth at all costs) and so I expect many subtle misalignments in a substantial integration.
Personally for these reasons I am more eager to see features developed in the LW codebase than the Manifold codebase.
2I would love to try having dialogues with people about Agent Foundations! I'm on the vaguely-pro side, and want to have a better understanding of people on the vaguely-con side; either people who think it's not useful, or people who are confused about what it is and why we're doing it, etc.
1Would love if expandable comment sections could be put on poll options. Most won't benefit, but some might benefit greatly, epsecially if it incentivized low word count.
For example: For EA being >50% of being net negative, I would like to make a short comment like:
I think EA is something like ~52% net negative and ~48% net positive, with wide error bars, because sign uncertainty is high, but EA being net neutral is arbitrarily close to 0%.
A better poll would ask if EA was >60% of being net negative.
non-extinction AI x-risk scenarios are unlikely
Many people disagreed with that. So, apparently many people believe that inescapable dystopias are not-unlikely? (If you're one of the people who disagreed with the quote, I'm curious to hear your thoughts on this.)
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?
I think the intelligence community ought to be watching AI risk, whether or not they actually are.
e.g. in the UK, the Socialist Workers Party was heavily infiltrated by undercover agents; widwly suspected at the time, subsequently officially confirmed. Now, you way well disagree with their politics, but it's pretty clear they didn't amount to a threat. Infiltration should probably have bailed out after quickly Establishing threat level was low.
AI risk, on the other hand ... from the point of view of an intelligence agency, there's uncertainty a...
Academia is sufficiently dysfunctional that if you want to make a great scientific discover(y) you should basically do it outside of academia.
I feel like this point is a bit confused.
A person believing this essentially has to have a kind of "Wherever I am is where the party's at" mindset, in which case he ought to have an instrumental view of academia. Like obviously, if I want to maximize the time I spend reading math books and solving math problems, doing it inside of academia would involve wasting time and is suboptimal. However, if my goal is to do ...
Rationalists would be better off if they tore up the old stuff, and started again from beginning.
21
Do you have a question you'd like to see argued about? Would you like to indicate your position and discuss it with someone who disagrees?
Add poll options to the thread below to find questions with lots of interest and disagreement.
How to use the poll
The goal is to show people where a lot of interesting disagreement lies. This can be used to find discussion and dialogue topics in the future.