The Alignment Forum should have more transparent membership standards

Thank you for the critique and sorry for the bad experience!

We should really create better notifications for people who apply, and set better expectations that you shouldn't expect a quick response (for reasons I will elaborate on below). I also overall think the current "become a member" experience is at least somewhat confusing and much less smooth than it could be, and this post helped me think about what we could do better.

Here are some thoughts and clarifications and responses to a bunch of the things you said.

I applied three times to the AF and the only way I could communicate with the AF was via a personal connection who was willing to inquire on my behalf. This is pretty unresponsive behavior from the AF team and suggests there are equity issues with access to the community.

Use the Intercom button in the bottom right corner! We really are very accessible and respond to inquiries usually within 24 hours, even during Christmas and other holiday periods! Probably the fact that you went via an intermediary (who pinged a LW team member who wasn't very involved in AIAF stuff on FB messenger) was exactly the reason why you had a less responsive experience. We really invest a lot in being accessible and responsive to inquiries, and we should maybe make it more clear that the Intercom is the recommended way to reach out to us (and that this will reliably get you to an AI Alignment Forum admin).

We are pretty transparent about the fact that we are generally very unlikely to accept almost any applications to the AI Alignment forum, and generally choose to expand membership proactively instead of reactively based on applications. In particular, the application is not something we respond to, it's something that we take into account when we do a proactive round of invites, which is something we could probably be clearer about. From the application text:

We accept very few new members to the AI Alignment Forum. Instead, our usual suggestion is that visitors post to LessWrong.com, a large and vibrant intellectual community with a strong interest in alignment research, along with rationality, philosophy, and a wide variety of other topics.
Posts and comments on LessWrong frequently get promoted to the AI Alignment Forum, where they'll automatically be visible to contributors here. We also use LessWrong as one of the main sources of new Alignment Forum members.
If you have produced technical work on AI alignment, on LessWrong or elsewhere -- e.g., papers, blog posts, or comments -- you're welcome to link to it here so we can take it into account in any future decisions to expand the ranks of the AI Alignment Forum.

This is also the reason why the moderator you talked to told you that they didn't want to make unilateral decisions on an application (you also reached out via Owen on like FB messenger to a member of the team only tangentially involved with AIAF stuff, which bypassed our usual processes that would have allowed us to properly escalate this). The right term for the applications might be more something like "information submission", that we take into account when deciding on who to invite, not something that will get a reply in a short period of time. We are very careful with invites and basically never invite anyone in whom we don't have a lot of trust, and who doesn't have a long track record of AI Alignment contributions that we think are good.

In general, membership to the AI Alignment Forum is much more powerful than "being allowed to post to the AI Alignment Forum". In particular, each member of the forum can promote any other comment or post to the AI Alignment forum.

As such, membership on the AI Alignment Forum is more similar to being a curator, instead of an author (most users with posts on the AI Alignment Forum do not have membership). Membership is only given to people we trust to make content promotion decisions, which is a high bar.

My profile is listed on this post, but I am actually not a member of the Alignment Forum, meaning I could not respond to any comments on the post (and cannot make other posts or comment on other posts).

This is a bit unfortunate and is a straightforward bug. Sorry about that! We currently have a system where the primary author of a post can always respond to comments, but we apparently didn't implement the same logic for coauthors. I will make a Pull Request to fix this tonight, and I expect this will be merged and live by the end of the week. Really sorry about that!

Also, coauthors currently just can't edit the post they are a coauthor on, which is mostly because there is some actual ambiguity on what the right solution here is, and the definitely-right thing would be a whole Google-Docs style permission system where you can give different people different levels of access to a post, and that is just a lot of development work. I do hope to have something like that eventually.

I do want to reiterate though that you can totally respond to any comments on your post. Just leave a comment on the LW version, and it will be promoted to the AIAF usually within a day, if it's relevant and high quality. That is the recommended way for non-members to respond to comments anywhere on the AIAF, and you can think of it like a public moderation queue.

(1) Why was AF closed to the public? This seems obviously bad for the community. We are excluding some number of people who would productively engage with safety content on the AF from doing so. Of course there should be some community standard (i.e. a "bar") for membership -- this is a separate concern. It could also be that some active LW-ers actually did move onto the AF over this time period, due to some proactive mods. But this is not a public process, and I would imagine there are still a bunch of false negatives for membership.

When talking to researchers when setting up the forum, we had almost universal consensus that they wouldn't want to participate in an open online forum, given the low quality of almost any open online forum on the internet. The closed nature of the forum is almost the full value-add for those researchers, who can rely on having a high signal-to-noise ratio in the comments and posts. The integration with LessWrong means that anyone can still comment, but only the high quality comments get promoted to the actual AI Alignment Forum, which allows filtering, and also makes people much more comfortable linking to AI Alignment Forum posts, since they can generally assume all content visible is high quality.

(2) I am also particularly concerned about anyone from the broader AI community finding out that this forum was effectively closed to the public, meaning closed to industry, academia, independent researchers, etc. The predominate view in the AI community is still that the (longtermist) AI Safety community holds fringe beliefs, by which I mean that job candidates on the circuit for professorships still refrain from talking about (longtermist) AI Safety in their job talks because they know it will lose them an offer (except maybe at Berkeley). I imagine the broader reaction to learning about this would be to further decrease how seriously AI Safety is taken in the AI community, which seems bad.

Since closed discussion venues and forums are the default for the vast majority of academic discussion, I am not super worried about this. The forum's adoption seems to me to have overall been a pretty substantial step towards making the field's discussion public. And the field overall also has vastly more of its discussion public than almost any academic field I can think of and can easily be responded to by researchers from a broad variety of fields, so I feel confused about the standard you are applying here. Which other fields or groups even have similar forums like this without very high standards for membership? And which one of the ones with membership display the comments publicly at all?

(3) I'm left wondering what the distinction is between the AF and LW (though this is less important). Is LW intended to be a venue for AI Safety discussion? Why not just make the AF that venue, and have LW be a hub for people interested in rationality, and have separate membership standards for each? If you're concerned about quality or value alignment, just make it hard to become an AF member (e.g. with trial periods). I think it is very weird for LW to be considered a stepping stone to the AF, which is how the mods were treating it. I can say that as a person in academia with a general academic Twitter audience, I did not want our interpretability review to appear only on LW because I think of it as a forum for discussing rationality and I think most newcomers would too.

The FAQ has some things on the relationship here, though it's definitely not as clear as it could be:

For non-members and future researchers, the place to interact with the content is LessWrong.com, where all Forum content will be crossposted.
[...]
Automatic Crossposting - Any new post or comment on the new AI Alignment Forum is automatically cross-posted to LessWrong.com. Accounts are also shared between the two platforms.
Content Promotion - Any comment or post on LessWrong can be promoted by members of the AI Alignment Forum from LessWrong to the AI Alignment Forum.

In short, content on the AI Alignment Forum is a strict subset of the content on LW. All content on the AI Alignment Forum is also on LW. This makes it hard to have separate standards for membership. The way the sites are set up, the only option is for the AIAF to have stricter standards than LW. Posting to the AI Alignment Forum currently also means posting to LessWrong. AI Alignment content is central to LessWrong and the site has always been one of the biggest online venues for AI Alignment discussion.

There are a lot of reasons for why we went with this setup. One of them is simply that online forums have a really hard time competing in the modern attention landscape (LessWrong is one of the longest lived and biggest online forums in the post-Facebook/Reddit era), and it's really hard to get people to frequently check multiple sites. Since a substantial fraction of AI Alignment Researchers we were most excited about were already checking LessWrong frequently, combining the two made it a lot easier to sustain a critical mass of participants. Previous attempts at AI Alignment online forums usually failed at this step, and I expect that without this integration, the AI Alignment Forum would have also never reached critical mass.

It also enabled a public commenting system that was still actively moderated and was much higher quality than the rest of the public internet. While LW comments are generally lower quality than AI Alignment Forum comments, they are still one of the highest quality comment sections on the internet, and allowing any external researchers to comment and post on there, and allow their comments and posts to be promoted and engaged with, enables a much more public form of discussion and research than a strictly closed forum.

We could have had something like submission queues and trial periods, but when we talked to researchers and online commenters who had engaged with systems like that, they had almost universally negative experiences. Trial systems require a very large amount of moderation effort, which we simply don't have available. If someone fails the trial, they are also now completely shut out from engaging with the content, which also seems quite bad. Submission queues also require a lot of moderation effort. They also create a much worse experience for the author, since they usually cannot receive any responses or votes or engagement until someone reviews their comment, which can sometimes take multiple days. In the current system, a commenter can immediately get an answer, or get votes on their comment, which also helps AIAF members decide whether to promote a comment.

(4) Besides the AI Safety vs. rationality distinction, there could be greater PR risks from a strong association between AI Safety and the LW community. LW has essays from Scott Alexander stickied, and though I really love his style of blogging, Scott Alexander is now a hot-button figure in the public culture war thanks to the New York Times. Broadly speaking, identifying as a rationalist now conveys some real cultural and political information. The big worry here would be if AI Safety was ever politicized in the way that, e.g., climate change is politicized -- that could be a huge obstacle to building support for work on AI safety. Maybe I’m too worried about this, or the slope isn’t that slippery.

Yeah, I think this is a pretty valid concern. This was one of the things we most talked to people about when starting the forum, and overall it seemed worth the cost. But I do agree that there is definitely some PR risk that comes from being associated with LW.

Overall, sorry for the less than ideal experience! I would also be happy to hop on a call if you want to discuss potential changes to the forum or the general AI Alignment research setup around LessWrong and the AI Alignment Forum. Just ping me via PM or Intercom and I would be happy to schedule something. Also, do really feel free to reach out to us via Intercom any time you want. I try to respond quickly, and often also have extended discussions on there, if we happen to be online at the same time.

[-]Peter Hase4y230

Thanks for the informative reply! This clarifies a lot about the forum to me, and I'm glad you found the post helpful in some ways.

Let me also add on to some of the points above.

Use the Intercom button in the bottom right corner!

This is good to know about! I simply never knew that was a chat button, and I guess Owen and our mod intermediary didn't know about it since it didn't come up? I bet we could have saved a lot of trouble if we'd first talked through this a few months ago.

In particular, the application is not something we respond to The right term for the applications might be more something like "information submission

Ok, also good to know. I can say that when I pressed "submit" for my "application," I expected it to get responded to. But it's fine to reframe what the form is.

We are very careful with invites and basically never invite anyone in whom we don't have a lot of trust, and who doesn't have a long track record of AI Alignment contributions that we think are good.

This is a more straightforward and more stringent criterion than anything in the application text. I know you all have an idea of what the application text means, but basically without an acceptance rate or examples of accepted "track record[s]," it's hard for anyone else to really know where the bar is. For the record, it's fine to have a bar this high. But is it worth trying to provide more information about acceptance, e.g. in terms of acceptance rates or examples of acceptable track records or something?

For example, does my track record satisfy the criteria here? I mean this sincerely and I don't mind whatever the answer is. I just want to try to make this clearer to people who are interested in the AF, since people can see my AF post, website, google scholar, this LW post, etc.

What would particularly be worth avoiding is the impression that this is an old boys' club, since membership depends so strongly on trust. Well, besides just avoiding the impression, it's good to ensure community access is equitable.

Membership on the AI Alignment Forum is more similar to being a curator, instead of an author

This is a huge shift in concepts in my mind! Definitely worth stating very clearly on the website and associated forms.

The closed nature of the forum is almost the full value-add for those researchers, who can rely on having a high signal-to-noise ratio in the comments and posts

I think we're using word "closed" in different ways. I used "closed" to mean that an arbitrary person on the internet had no means of becoming an AF member (on the grounds that "applications" were not being processed and a mod told us that no one was currently responsible for promotions). I think you're using "closed" to mean "has-a-really-high-bar-for-content-visibility." I think we both agree these are different dimensions a forum can pick its position on, but it seems to be a source of confusion at the moment. Of course a high signal-to-noise ratio is good and something the AF should aim for. And I think you're suggesting that there might not be many false negatives due to integration with LW, which is possible but I'm pretty uncertain about the quantities of false negatives / false positives here (and see more on LW integration below).

Since closed discussion venues and forums are the default for the vast majority of academic discussion, I am not super worried about this. The forum's adoption seems to me to have overall been a pretty substantial step towards making the field's discussion public.

Per my definition of closed, no academic discussion is closed, because anyone in theory can get a paper accepted to a journal/conference, attend the related meaning, and participate in the discourse. I am not actually talking about visibility to the broader public, but rather the access of any individual to the discourse, which feels more important to me.

Per your definition of closed, it could be that the AF has a higher or lower bar-for-content-visibility than academic venues. I think I had an easier time getting one of my safety papers into an AI conference and discussing it with other researchers than getting my post onto the AF (in terms of ease of submission and engagement).

I agree that the AF has brought AI Safety as a topic further into the public discourse than it would have been otherwise, by means of visibility and growing the number of people involved in the discourse, but this does not mean AF has made membership in the community any more public than it would have been otherwise (which is definitional and not semantic point).

What I still worry about is that I predict other researchers would react negatively if I told them "oh yeah there's this great AI Safety forum on the internet, but you can't post any of your ideas there or comment on other people's ideas, except by posting on a rationality forum and hoping one of the AF members or mods promotes your post to the safety forum. What is rationality? Oh it's this other community that tries to..."

As Rohin also suggested, it seems to make a difference whether you treat the reference class for the AF as the journal/conference proceedings or the conversations around a given paper poster.

In short, content on the AI Alignment Forum is a strict subset of the content on LW.

A lot of the reasons for this make sense to me, especially since it also seems plausible to me that "without this integration, the AI Alignment Forum would have also never reached critical mass."

If someone fails the trial, they are also now completely shut out from engaging with the content, which also seems quite bad.

But is it worse than never having a trial to begin with? Right now people are shut out by default from AF, except by going through LW (and see below).

In the current system, a commenter can immediately get an answer, or get votes on their comment, which also helps AIAF members decide whether to promote a comment.

So it seems like the greatest current advantage of the existing setup, in terms of mod workload, is that you're crowdsourcing the AF moderation/content promotion via LW.

First let me note that it still seems relatively unclear what kind of content standards there are for getting things onto AF via LW, and it is unclear what kinds of recourse are available. Do you have to hope that an AF member or mod sees the post and thinks it is "relevant and high quality"? Are there any more specific criteria for post promotion? Is there anything more proactive a poster can do than post and wait? What should a user do if they thought a post/comment should be promoted and it wasn't? Even after your assurance that the relevant people tend to get to LW posts/comments within a day to decide what to do with them, I still feel uncertain about how easily I could engage with AF content in this manner. I think my crux here is how passive this feels. It's mainly a waiting-and-hoping game from the LW side.

Second, I understand that LW fulfilled the crowdsourcing role here naturally at the inception of the AF. I think it's worth revisiting whether it best fulfills this role going forward. It seems like we both have PR concerns, for instance. Ignoring engineering and UI considerations for a second, why not split the AF into two channels: a channel for current members and a channel for new posters (where content from the New channel kept distinct from the main channel)? This way content promotion for AF is managed by AF members visiting AF and not by AF members visiting LW. This seems like it would create more value/culture alignment for primary AF content. Or maybe you want the larger, tangentially aligned LW community to help with crowdsourcing, due to the advantage of size in crowdsourcing. But then why not grow the AF ranks to solve that problem?

Let me end by re-emphasizing that I think one of my cruxes here is whether non-members are passive or active in their participation in the AF. Right now the process feels very passive from my end, and I do not know what kind of recourse is available when I want to participate in AF dialogue but am not allowed to. Whatever the standards and rules end up being, the whole process feels a lot better if I, e.g., can explicitly request that posts/comments are promoted, and I can get feedback on why posts/comments are not promoted. But maybe that just turns into peer review, and maybe crowdsourcing is the more efficient solution for achieving all the things we want.

Lastly, thanks for offering the call! Though it may be slower to type I think it could be beneficial to keep all the discussion here (if anyone has any more thoughts to chip in).

[-]habryka4y180

Second, I understand that LW fulfilled the crowdsourcing role here naturally at the inception of the AF. I think it's worth revisiting whether it best fulfills this role going forward. It seems like we both have PR concerns, for instance. Ignoring engineering and UI considerations for a second, why not split the AF into two channels: a channel for current members and a channel for new posters (where content from the New channel kept distinct from the main channel)? This way content promotion for AF is managed by AF members visiting AF and not by AF members visiting LW. This seems like it would create more value/culture alignment for primary AF content. Or maybe you want the larger, tangentially aligned LW community to help with crowdsourcing, due to the advantage of size in crowdsourcing. But then why not grow the AF ranks to solve that problem?

I do think it's worth reconsidering this from time to time. Here are some thoughts.

I am still worried that the AIAF would lose its critical mass if it were to become a completely separate website.

I also think the LW culture is a healthy context in which to do AI Alignment research, though it of course has many flaws, and I am wary of cutting that connection.

But the biggest obstacle is probably just operational capacity. Because of the way the AIAF and LW share code and data, it drastically reduces the engineering and moderation complexity, and the AIAF/LW team is currently already kind of swamped with all the different things we want to do. I have been trying to expand the team and hire more people, but it's hard and I've already been struggling to manage a team of the current size. I am really putting a lot of my effort into figuring out how to gain more operational capacity for the organization, but we are definitely somewhat bottlenecked on my ability to manage a larger organization and finding good people that I would trust to handle this job.

I would love a small dedicated team that could do much more ambitious things with the AI Alignment Forum, and am working towards that, at which point I think changing the relationship between LW and the AIAF is much more on the table. But before that happens, I think it's unlikely we should make a big push to change this relationship drastically.

[-]habryka4y160

This is good to know about! I simply never knew that was a chat button, and I guess Owen and our mod intermediary didn't know about it since it didn't come up? I bet we could have saved a lot of trouble if we'd first talked through this a few months ago.

The mod intermediary (Ben Pace) definitely knew, and Owen has also messaged us quite a few times on Intercom in the past, so not sure what exactly went wrong here. Looking at the chat logs, I think it was mostly a failure of process by the LW team, caused by FB messenger being such an informal chat-tool that's usually used non-professionally. So I think Ben treated the requests more as a random informal chat among friends instead of a thing to pay a lot of attention to (and he was also particularly busy with two non-AIAF projects both times he was messaged by Owen).

For example, does my track record satisfy the criteria here? I mean this sincerely and I don't mind whatever the answer is. I just want to try to make this clearer to people who are interested in the AF, since people can see my AF post, website, google scholar, this LW post, etc.

We did indeed think about whether you would meet the bar, and decided you don't currently. Though the post with Owen was great. I think we would likely try to find some references and ask some other people in the field if they got value out of your work to make a final call here, but I think at the moment we wouldn't give you membership without some additional highly-relevant papers or really great references.

I do think we might have a particular blindspot around transparency work, which is something I've been wanting to fix. I haven't looked much into the state of the literature on transparency, and since it's much easier to publish transparency work in existing CS journals than some of the more agent-foundation-like stuff, and there is more of an existing field to engage with, a lot of things happening have been happening somewhat out of my view (and I think also the other mod's views). I recently updated upwards on the importance of transparency work for AI Alignment, and I think it's pretty plausible our policy on this dimension might change, and that after reorienting towards that, I will feel really silly for thinking you didn't meet the bar.

I think we're using word "closed" in different ways. I used "closed" to mean that an arbitrary person on the internet had no means of becoming an AF member (on the grounds that "applications" were not being processed and a mod told us that no one was currently responsible for promotions).

But clearly the relevant comparison isn't "has no means of becoming an AF member". The bar should be "has no means of submitting a paper/post/comment", and the bar for that is pretty low for the AIAF. I recognize that there is some awkwardness in commenting or posting first to LessWrong, and there are some issues with perceived reliability, but the bar is overall substantially lower than for most journals, and much easier to clear. As I mentioned above, AF membership itself doesn't have a great equivalent for traditional journals or conferences, but is overall a bit more like "being on the conference committee" or "being a reviewer for the conference" or "being an editor for the journal", all of which are much more opaque and harder to get access to, at least in my experience. Just submitting to the AIAF is definitely a bit confusing, but I don't think overall harder than submitting to a journal.

I am not actually talking about visibility to the broader public, but rather the access of any individual to the discourse, which feels more important to me.

I think I am most confused what you mean by "access to the discourse". Again, if you comment on LessWrong, people will respond to you. Journal's don't usually have any kind of commenting system. AI in particular has a lot of conferences, which helps, though I think most people would still think that getting to one of those conferences and making connections in-person and participating in discussion there is overall a much higher bar than what we have with the LW/AIAF integration. Of course, many people in CS will have already cleared a lot of those bars since they are used to going to conferences, but as a non-academic the bar seems much higher to me.

But is it worse than never having a trial to begin with? Right now people are shut out by default from AF, except by going through LW (and see below).

We discussed this a lot when setting up the forum. My current honest guess is that no trial is indeed better than having a trial, mostly because people do really hate having things taken away from them, and almost everyone we talked to expected that people whose trial would end without membership would quite reliably feel very disappointed or get angry at us, even when we suggested framings that made it very clear that membership should not be expected after a trial.

I also share this from experience. When I did internships where it was made clear to me that I was very unlikely to get a job offer by the end of it, I still felt a really strong sense of disappointment and frustration with the organization when I didn't end up getting an offer. I simply had invested so much by that point, and getting angry felt like a tempting strategy to push the odds in my favor (I never ended up taking much action based on that frustration, mostly realizing that it felt kind of adversarial and ungrounded, but I expect others would not behave the same way, and having people on the internet scream at you or try to attack you publicly is quite stressful and can also make others much less interested in being part of the forum).

I think my crux here is how passive this feels. It's mainly a waiting-and-hoping game from the LW side.

I do think the right thing we should tell people here is to post to LW, and if after a day it hasn't been submitted, to just ping us on Intercom, and then we can give you a straightforward answer on whether it will be promoted within 24 hours. I respond to something like 1-2 requests like this a month, and I could easily handle 10x more, so making that path easier feels like it could make the whole process feel more active.

So it seems like the greatest current advantage of the existing setup, in terms of mod workload, is that you're crowdsourcing the AF moderation/content promotion via LW.

I don't think this is exactly true. It's more that LW is trying to provide an "OK" space for people to submit their AIAF content to that comes with immediate visibility and engagement, and it's trying to set expectations about the likelihood of content getting promoted. But yeah, the crowdsourced moderation and content promotion is also nice.

can explicitly request that posts/comments are promoted, and I can get feedback on why posts/comments are not promoted.

I think this expectation of feedback is really the primary reason we can't offer this. Providing feedback for every single piece of content is a massive amount of moderation work. We are talking about 2-3 fulltime people just on this job alone. The only option we would have is to have a system that just accepts and rejects comments and posts, but would do so without any justification for the vast majority of them. I expect this would reliably make people feel angry and even more powerless and then we would have even more people feel like they have no idea how to interact with us, or bang on our door to demand that we explain why we didn't promote a piece of content.

There is a good reason why there basically exist no other platforms like the AI Alignment Forum on the internet. Content moderation and quality control is a really hard job that reliably has people get angry at you or demand things from you, and if we don't put in clever systems to somehow reduce that workload or make it less painful, we will either end up drastically lowering our standards, or just burn out and close the forum off completely, the same way the vast majority of similar forums have in the past.

I mean, of course, there are probably clever solutions to this problem, but I do think they don't look like "just have a submission queue that you then accept/reject and give feedback on". I think that specific format, while common, also has reliable failure modes that make me very hesitant to use it.

[-][anonymous]4y100

Just chiming in here to say that I completely forgot about Intercom during this entire series of events, and I wish I had remembered/used it earlier.

(I disabled the button a long time ago, and it has been literal years since I used it last.)

[-]habryka4y20

Checks out. I remember getting some messages from you on there, but upon checking, that was indeed like 3 years ago.

[-]Peter Hase4y50

But the biggest obstacle is probably just operational capacity.

I see. I know the team has its limits and has already been in a lot of work to propping up AF/LW, which is generally appreciated!

I think I am most confused what you mean by "access to the discourse".

I mean the ability to freely participate in discussion, by means of directly posting and commenting on threads where the discussion is occurring. Sorry for not making this clearer. I should have more clearly distinguished this from the ability to read the discussion, and the ability to participate in the discussion after external approval.

But clearly the relevant comparison isn't "has no means of becoming an AF member". The bar should be "has no means of submitting a paper/post/comment"

Yeah let me try to switch from making this about the definition of "closed" to just an issue about people's preferences. Some people will be satisfied with the level of access to the AF afforded to them by the current system. Others will not be satisfied with that, and would prefer that they had direct/unrestricted access to the AF. So this is an interesting problem: should the AF set a bar for direct/unrestricted access to the AF, which everyone either meets or does not meet; or should the AF give members direct access, and then given non-members access to the AF via LW for specific posts/comments according to crowdsourced approval or an AF member's approval? (Of course there are other variants of these). I don't know what the best answer is, how many people's preferences are satisfied by either plan, whose preferences matter most, etc.

My current honest guess is that no trial is indeed better than having a trial

I can see why, for the reasons you outline, it would be psychologically worse for everyone to have trials than not have trials. But I think this is a particularly interesting point, because I have a gut-level reaction about communities that aren't willing to have trials. It triggers some suspicion in me that the community isn't healthy enough to grow or isn't interested in growing. Neither of these concerns is necessarily accurate — but I think this is why I predict a negative reaction from other researchers to this news (similar to my original point (2)). Typically people want their ideas to spread and want their ideology to be bolstered by additional voices, and any degree of exclusivity to an academic venue raises alarm bells in my mind about their true motives / the ideological underpinnings of their work. Anyway, these are just some negative reactions, and I think, for me, these are pretty well outweighed by all the other positive inside-view aspects of how I think of the AI safety community.

I do think the right thing we should tell people here is to post to LW, and if after a day it hasn't been submitted, to just ping us on Intercom, and then we can give you a straightforward answer on whether it will be promoted within 24 hours.

Great!

The only option we would have is to have a system that just accepts and rejects comments and posts, but would do so without any justification for the vast majority of them.

Sorry, isn't this the current system? Or do you mean something automated? See next comment, which I left automation out from. Right now the promotion system is a black-box from the user's end, since they don't know when AF members are looking at posts or how they decide to promote them, in the same way that an automatic system would be a black-box system to a user if they didn't know how it worked.

There is a good reason why there basically exist no other platforms like the AI Alignment Forum on the internet. Content moderation and quality control is a really hard job that reliably has people get angry at you or demand things from you, and if we don't put in clever systems to somehow reduce that workload or make it less painful, we will either end up drastically lowering our standards, or just burn out and close the forum off completely, the same way the vast majority of similar forums have in the past.

Yeah, and this is a problem every social media company struggles with, so I don't want to shame the mod team for struggling with it.

But I do want to emphasize that it's not a great state to be in to have no recourse systems. Every forum mod team should provide recourse/feedback in reasonable proportion to its available resources. It seems like you're predicting that users would feel angry/powerless based on a kind of system with limited recourse/feedback, and hence everyone would be worse off with this system. I think something else must occur: without any recourse, the level of anger+powerlessness is high, and as more recourse is added, the amount of these feelings should decline. I think this should happen as long as user expectations are calibrated to what the recourse system can provide. If the forum moves from "no reason for non-promotion, upon request" to "one-sentence reason for non-promotion (and no more!), upon request", people might complain about the standard but they shouldn't then feel angry about only getting one sentence (in the sense that their expectations are not being violated, so I don't think they would be angry). And if users are angry about getting a one-sentence-reason policy, then wouldn't they be angrier about a no-reason-policy? As long as expectations are set clearly, I can't imagine a world where increasing the amount of recourse available is bad for the forum.

Maybe this would be a good point to recap, from the mod team's perspective, what are some ways the AF+LW could more clearly set user expectations about how things work. I think it would also be valuable to specify what happens when things don't go how users want them to go, and to assess whether any reasonable steps should be taken to increase the transparency of AF content moderation. No need to re-do the whole discussion in the post+comments (i.e. no need to justify any decisions) — I just want to make sure this discussion turns into action items as the mods think are appropriate.

[-]habryka4y50

Sorry, isn't this the current system? Or do you mean something automated? See next comment, which I left automation out from.

Sorry, I was suggesting a system in which instead of first posting to LW via the LW interface, you just directly submit to the AIAF, without ever having to think about or go to LW. Then, there is a submission queue that is only visible to some moderators of the AIAF that decides whether your content shows up on both LW and the AIAF, or on neither. This would make it more similar to classical moderated comment-systems. I think a system like this would be clearer to users, since it's relatively common on the internet, but would also have the problems I described.

It seems like you're predicting that users would feel angry/powerless based on a kind of system with limited recourse/feedback, and hence everyone would be worse off with this system.

One specific problem with having a submission + admin-review system is that the user has to invest a lot of resources into writing a post, and then only after they invested all of those resources do they get to know whether they get any benefit from what they produced and whether their content (which they might have spent dozens of hours writing) is accepted. This is I think one of the primary things that creates a lot of resentment, and when I talk to people considering publishing in various journals, this is often one of the primary reasons they cite for not doing so.

When designing systems like this, I try to think of ways in which we can give the user feedback at the earliest level of investment, and make incremental benefit available as early as possible. The current system is designed that even if your post doesn't get promoted to the AIAF, you will likely still get some feedback and benefit from having it on LW. And also, it tries to set expectations that getting a post onto the AIAF is more like a bonus, and the immediate level of reward to expect for the average user, is what you get from posting on LW, which in my experience from user-interviews causes people to publish earlier and faster and get more feedback before getting really invested, in a way that I think results in less resentment overall if it doesn't get promoted.

I do think some people see very little reward in posting to LW instead of the AIAF, and for those this system is much worse than for the others. Those users still feel like they have to invest all of this upfront labor to get something onto the AIAF, and then have even less certainty than a normal submission system would provide on whether their content gets promoted, and then have even less recourse than a usual academic submission system would provide. I think it is pretty important for us to think more through the experience of those users, of which I think you are a good representative example.

Maybe this would be a good point to recap, from the mod team's perspective, what are some ways the AF+LW could more clearly set user expectations about how things work. I think it would also be valuable to specify what happens when things don't go how users want them to go, and to assess whether any reasonable steps should be taken to increase the transparency of AF content moderation. No need to re-do the whole discussion in the post+comments (i.e. no need to justify any decisions) — I just want to make sure this discussion turns into action items as the mods think are appropriate.

I am still thinking through what the right changes we want to make to the system are, but here is a guess on a system that feels good to me:

We do a trial where non-AF members get a button for "submit a comment to the AIAF" and "submit a post to the AIAF" when they log into the alignmentforum.org website
When they click that button a tiny box shows up that explains the setup of posting to the AIAF to them. It says something like the following:
- "When you submit a comment or post to the AI Alignment Forum two things happen:
  - The post/comment is immediately public and commentable on our sister-platform LessWrong.com, where researchers can immediately provide feedback and thoughts on your submission. You can immediately link to your submission and invite others to comment on it.
  - The post/comment enters a review queue that is reviewed within three business days by an admin on whether to accept your submission to the AI Alignment Forum, and if it does not get accepted, the admin will provide you with a short one-sentence explanation for why they made that decision. The admin uses the discussion and reaction on LessWrong to help us judge whether the content is a good fit for the AI Alignment Forum.
- The AI Alignment Forum admins are monitoring all activity on the site, and after you participated in the discussion on the AI Alignment Forum and LessWrong this way, an admin might promote you to a full member of the AI Alignment Forum, who can post to the forum without the need for review, and who can promote other people's comments and posts from LessWrong.com to the AI Alignment Forum. If you have questions about full membership, or any part of this process, please don't hesitate to reach out to us (the AIAF admins) via the Intercom in the bottom right corner of the forum."
When you finish submitting your comment or post you automatically get redirected to the LW version of the corresponding page where you can see your comment/post live, and it will show (just to you) a small badge saying "awaiting AI Alignment Forum review"

I think we probably have the capacity to actually handle this submission queue and provide feedback, though this assumption might just turn out to be wrong, in which case I would revert those changes.

Alternatively, we could provide an option for "either show this content on the AIAF, or show it nowhere", but I think that would actually end up being kind of messy and complicated, and the setup above strikes me as better. But it does point people quite directly to LessWrong.com in a way that strengthens the association between the two sites in a way that might be costly.

[-]ChristianKl4y20

The post/comment enters a review queue that is reviewed within three business days by an admin on whether to accept your submission to the AI Alignment Forum

If you believe in Alignment Forum participants making review judgements, how about using a review queue that works more like StackOverflow for this then admin labor?

I would expect a system that allows Alignment Forum participants to work through the queue to lead to faster reviews and be more easy to scale.

[-]habryka4y20

My general philosophy for things like this is "do it in-house for a while so you understand what kinds of problems come up and make sure you have a good experience. After you really have it down maybe consider outsourcing it, or requesting volunteer labor for it."

So I think eventually asking for a more crowdsourced solution seems reasonable, though I think that would come a few months after.

[-]Gordon Seidoh Worley4y160

Does much curation actually happen, where members of the forum choose to promote comments and posts to AF? I've occasionally done this for comments and never for posts (other than my own), though I do sometimes hit the "suggest for AF" button on a post I think should be there but am not so confident as to make a unilateral decision. So I was surprised by your comments about curation because I don't much think of that as an activity AF forum members perform.

[-]habryka4y20

As far as I can tell the actual level of curation by different members is very heavy-tailed. So a few users do almost all of it, most users do very little. Though since every member has the potential to do a lot of promoting comments and moving posts, it's still a bar to be careful with.

[-]Neel Nanda4y130

And the field overall also has vastly more of its discussion public than almost any academic field I can think of and can easily be responded to by researchers from a broad variety of fields

What do you mean by this? I imagine the default experience of a researcher who wants to respond to some research but has minimal prior exposure to the community, is to be linked to the Alignment Forum, try to comment, and not be able to. I expect commenting on LessWrong to be non obvious as a thing to do, and to feel low-status/not like having a real academic discussion

[-]habryka4y40

Compared to just LW, it is more of a hassle. But most comments by academics will indeed be promoted quickly to the AI Alignment Forum, and it's much less hassle than commenting on an academically published paper (which is close to impossible).

There are barriers to commenting and engaging, but overall it just seems to me that the barriers are much lower than commenting on journal and conference articles, which feels like the relevant reference class.

[-]Neel Nanda4y60

Hmm, fair point. I feel concerned at how illegible that is though, especially to an academic outsider who wants to engage but lacks context on LW. Eg, I've been using AF for a while, and wasn't aware that comments were regularly promoted from LW. And if we're talking about perception of the field, I think surface level impressions like this are super important

[-]habryka4y40

Yeah, I do think we should just really make things clearer here. Managing the exact extent of the relationship between LW and AIAF is a bit of a delicate balance, but I do think the current communication around it is definitely not ideal.

[-]ChristianKl4y60

The AI Alignment Forum currently has a New Comment field at the bottom when a user isn't logged in that has a submit button. A researcher who visits the alignment forum and types out his comment has a reasonable expectation to when he clicks submit there will be a process that leads to publishing of the comment. That field sets the wrong expecations about what happens when a user who visits the alignment forum registers an account.

If I login then I get shown "You must be approved by an admin to comment on the AI Alignment Forum" and not get told anything about the fact that I might post comments to AI Alignment Forum articles on LW.

This setup is confusing for a new user who understands how journal and conference articles work but who doesn't understand how the alignment forum works.

I think it would make sense to completely remove the New Comment field when a user isn't logged in to not give the appearance that it's easy to comment directly on the alignment forum.

Instead of the New Comment field the sentence "You must be approved by an admin to comment on the AI Alignment Forum" could be displayed when a user isn't logged in as well. That message might also be worded friendlier and contain a link to a page that explains the setup.

[-]habryka4y40

I think it would make sense to completely remove the New Comment field when a user isn't logged in to not give the appearance that it's easy to comment directly on the alignment forum.

Yep, sorry, this is the expected behavior and anything else is a bug. It used to be this way, but some changes to some of the commenting code must have broken it. Definitely our bad.

[-]Neel Nanda4y20

A similar bug - when I go to the AF, the top right says Log In, then has a Sign Up option, and leads me through the standard sign-up process. Given that it's invite only, seems like it should tell people this, and redirect them to make a LW account?

[-]habryka4y20

I do think "non-member AIAF" account is a useful abstraction, just one we haven't leveraged very much. But, for example, it does allow you to respond to comments on your post on the AIAF, even if you are not a member. So the signup does make sense.

[-]CraigMichael4y80

In short, content on the AI Alignment Forum is a strict subset of the content on LW. All content on the AI Alignment Forum is also on LW… There are a lot of reasons for why we went with this setup.

This makes sense now that you’ve explained it.

It has been confusing for me since I signed up on LW and would be cool if it was spelled out a bit more obviously.

[-]Steven Byrnes4y60

The integration with LessWrong means that anyone can still comment

Speaking of this, if I go to AF without being logged in, there's a box at the bottom that says "New comment. Write here. Select text for formatting options... SUBMIT" But non-members can't write comments right? Seems kinda misleading... Well I guess I just don't know: What happens if a non-member (either LW-but-not-AF member or neither-AF-nor-LW member) writes a comment in the box and presses submit? (I guess I could do the experiment myself but I don't want to create a test comment that someone then has to then go delete.)

To work around this (apparent?) issue, I was planning to start ending some of my AF posts (those where I expect significant non-AF readership) with a line like:

[Email me] (mailto link) or [comment on the lesswrong crosspost] (LW link)

I think this will work, but only if I obfuscate the LW link using bitly or whatever, because by default LW links are auto-converted to AF links when viewed on AF, right? I haven't tried it yet.

(Sorry to shoehorn my general AF meta questions into this thread.)

[-]habryka4y40

Oops, yeah, this just seems like a straightforward bug. When you press "Submit" it asks you to log-in, and when you then log-in with a non-member account, the box just disappears and there is no obvious way to get your written content back. That seems like a terrible experience. I will fix that, I think we introduced this behavior when we made some changes on LW to how unsubmitted comments are saved.

Talking about general AIAF meta question, an option I've been considering for a while is to have a submission queue for AIAF non-members on the site, where they can submit posts to the AIAF directly, without going through LessWrong. The big concern here is that someone would have to review all of them, and also that I would want most of them to be rejected since they aren't a good fit for the forum, and this seems more likely to make people unhappy than asking people to post to LW first.

I think the current setup is the better choice here, since I am worried the submission queue would cause a bunch of people to spend a lot of time writing posts, and then get told they won't be accepted to the forum and that they wasted a lot of time, which is a much worse experience than being told very early that they should just post to LW and then ask for it being promoted (which I think sets better expectations). But I would be curious if people have different takes.

[-]Dustin4y80

As a full-stack developer (a term which I kinda hate!) I just want to take this opportunity to mention to people who happen to read this that bug reports can be very helpful!

For any potential bug like Steven Byrnes mentions, there can be many people who experienced it but didn't report it. With small teams of developers and complicated sites there can be many types of bugs that irritate many users but no one who can fix the bugs ends up finding out about the problem for a long time.

I know that in general bug reports can often feel like yelling into the void, but in my experience, they're almost always getting considered by someone.

[-]habryka4y20

Thank you! Indeed, all bug reports are greatly appreciated! If you really ever notice anything that bothers you, just send a quick message on Intercom. We won't always get around to fixing all bugs, but we use bug reports also as one of our primary prioritization tools, and this specific bug did just go unnoticed for a long time.

[-]Rohin Shah4y*40

Since closed discussion venues and forums are the default for the vast majority of academic discussion, I am not super worried about this. The forum's adoption seems to me to have overall been a pretty substantial step towards making the field's discussion public. And the field overall also has vastly more of its discussion public than almost any academic field I can think of and can easily be responded to by researchers from a broad variety of fields, so I feel confused about the standard you are applying here. Which other fields or groups even have similar forums like this without very high standards for membership? And which one of the ones with membership display the comments publicly at all?

While I agree with you on the object-level claims, I don't think this is how it would be perceived. By default academics work on projects that (eventually) get published in conferences / journals (and in CS these are often free to access). If you see AIAF as part of the same reference class as conferences / journals, then AIAF is certainly much more closed in terms of its explicit policies.

(The object-level counterarguments are (1) AIAF is more like the conversations that researchers have with each other when attending a conference, which are typically private and (2) while conferences in theory have double blind reviews, it is usually the case that in practice you can't contribute to the conversation without following a set of norms of the field, many of which are only tangentially related to quality and only known to existing academics. But (1) is not obvious to people who don't follow AIAF much, and is in fact false of OP's post in particular (which really is quite close to an academic literature review / survey article that could be published) and (2) is not commonly believed / accepted.)

[-]habryka4y50

If you see AIAF as part of the same reference class as conferences / journals, then AIAF is certainly much more closed in terms of its explicit policies.

I am a bit confused by this. It seems that submitting a post to the AIAF is much easier than submitting a post to a journal, and is much less of a hassle. So in what sense is it much more closed?

[-]Rohin Shah4y70

Only members can submit a post to the AIAF, whereas anyone can submit to a conference / journal?

(Maybe you count "writing a post on LessWrong" as a "submission" of a post to the AIAF, that's not how I would expect it to be perceived by default, and not how I perceived it prior to this post.)

[-]habryka4y60

Yeah, I would count writing a LW post plus maybe messaging an admin about it as the equivalent of "the barrier to participating in the discourse". We don't have a submission system as such, so of course it isn't exactly equivalent to submission, but the overall barrier to participation strikes me as substantially lower.

[-]Dustin4y20

To be clear...now having read the post and comments you do not consider it more closed?

[-]Rohin Shah4y140

I feel like we should taboo "closed".

Both before and after reading the post, I think that AIAF caused AI alignment discussion to be much more publicly readable (relative to academia).

After reading the post / comments, I think that the AIAF is more publicly writable than academia. Before reading the post / comments, I did not think this -- I wouldn't have said that writing a post on LW was "submitting" to the AIAF, since it didn't seem to me like there were people making a considered decision on whether to promote LW posts to AIAF.

Both before and after reading the post, I think that the AIAF will not be perceived (at least by academics) to be more publicly writable than academia.

[-]ChristianKl4y20

It seems to me like the main problem is that AIAF currently does a bad job at giving a naive visitor an idea about how it's setup works. Do you think that a better explanation on the side of AIAF would solve the issue or do you believe it to be deeper?

[-]adamShimi4y40

I'm still confused by half the comments on this post. How can people be confused by a setting explained in detail in the only post always pinned in the AF, which is a FAQ?

[-]Neel Nanda4y90

I think most people just don't read the manual? And I think good user interfaces don't assume they do

Speaking personally, I'm an alignment forum member, read a bunch of posts on there, but never even noticed that post existed

[-]adamShimi4y60

I want to push back on that. I agree that most people don't read the manual, but I think that if you're confused about something and then don't read the manual, it's on you. I also don't think they could make it much more obvious than being always on the front page.

Maybe the main criticism is that this FAQ/intro post has a bunch of info about the first AF sequences that is probably irrelevant to most newcomers.

[-]ChristianKl4y170

It would for example be possible to have a notice at the bottom of alignment forum pages to user that aren't locked in that says: "If you aren't a member of the alignment forum and want to comment on this post, you can do so at [link to LessWrong post]. Learn more [link to FAQ]"

Such a link would strengthen the association between LessWrong and AIAF for a naive user that reads a AIAF posts. There might be drawbacks for strengthen that association but it would help the naive user to get the idea that the way to interact with AIAF posts for non-AIAF members is through LessWrong.

I agree that most people don't read the manual, but I think that if you're confused about something and then don't read the manual, it's on you.

The goal of AIAF is to be well accepted by the AI field. If people from that field come to AIAF and have a lesser opinion of AIAF because they don't really understand how it works, you can say that's on them but it's still bad for AIAF.

[-]habryka4y20

Yeah, I was proposing something like this in this comment response to Peter.

[-]Neel Nanda4y70

I agree that most people don't read the manual, but I think that if you're confused about something and then don't read the manual, it's on you.

I think responsibility is the wrong framing here? There are empirical questions of 'what proportion of users will try engaging with the software?', 'how many users will feel confused?', 'how many users will be frustrated and quit/leave with a bad impression?'. I think the Alignment Forum should be (in part) designed with these questions in mind. If there's a post on the front page that people 'could' think to read, but in practice don't, then I think this matters.

I also don't think they could make it much more obvious than being always on the front page.

I disagree. I think the right way to do user interfaces is to present the relevant information to the user at the appropriate time. Eg, when they try to sign-up, give a pop-up explaining how that process works (or linking to the relevant part of the FAQ). Ditto when they try making a comment, or making a post. I expect this would exposure many more users to the right information at the right time, rather than needing them to think to look at the stickied post, and filter through for the information they want

[-]Dustin4y20

I think part of the problem is that it's not always obvious that you're confused about something.

If you don't know that the UI has led you to make wrong assumptions about the way it works, you won't even know to go look at the manual.

(Also, as someone who has designed lots of UI's...for many types of UI's, if the user has to go look at the manual it means I've got something to improve in the UI.)

[-]Rohin Shah4y20

I think giving people better beliefs about how the AIAF works would probably solve the issue, though that doesn't necessarily come from better explanations, e.g. I much prefer things like your suggestion here, where you're providing some info at exactly the time it is relevant, so that people actually read it. (Perhaps that's what you mean by "better explanations".)

[-]abramdemski4y130

As someone who had some involvement in the design of the alignment forum, and the systems you are describing, this post has given me a lot to think about. I think it is clear that the current system is not working. However, one point you made seems rather opposed to my picture, and worth discussing more:

The AF being closed to the public is bad for the quality of AF discourse.

The reason, in my mind, for the current system is because it's difficult to know who is and is not suitable for membership. Therefore, we have a very opaque and unpredictable membership standard, "by necessity". In an attempt to compensate, we try to make membership matter as little as possible, IE we hope non-members will feel comfortable posting on LessWrong and then have their posts get promoted. Obviously this is not working very well at all, but that was my hope: (A) keep membership standards very high and therefore opaque, (B) make membership matter as little as possible.

Why do I think standards should be very high?

My model is that if the alignment forum were open to the public, there would be a lot more very low-quality posts and (especially) comments. I could be wrong here -- standards of comments on lesswrong are pretty good -- but I worry about what happens if it's fine for a while and then (perhaps gradually) there starts to be a need to close access down more to maintain quality. Opening things up seems a lot easier than closing them down again, because at that point there would be a need to make tough decisions.

So I'm left wondering what to do. Can we keep standards high, while also making them transparent? I don't know. But it does seem like something needs to change.

[-]abramdemski4y60

After reading more of the discussion on this post, I think my reply here was conflating different notions of "open" in a way which has been pointed out already in comments.

[-]abramdemski4y40

I'm currently fairly swayed that the alignment forum should be more "open" in the sense Peter intends:

Per my definition of closed, no academic discussion is closed, because anyone in theory can get a paper accepted to a journal/conference, attend the related meaning, and participate in the discourse. I am not actually talking about visibility to the broader public, but rather the access of any individual to the discourse, which feels more important to me.

However, I am not sure how to accomplish this. (Specifically, I am not sure how to accomplish this without too much added work, and maintaining other properties we want the forum to have.)

LESSWRONG
LW

LESSWRONG
LW

68

The Alignment Forum should have more transparent membership standards

68

68