Amalthea

I think it's unfortunate that you insist on giving them credit for their weasely behavior in the NS story. (I would similarly object to someone repeatedly citing that as egregiously bad behavior tbc.). They clearly didn't try very hard to do the right thing here and to the extent that they didn't optimize for their PR, that was more due to incompetence or the internal mathematicians wanting to be part of the story and acting unwisely etc.

I think the Anthropic employee at least had access to a large number of excess tokens and potentially internal models so we shouldn't be treating them like "just a guy". (I find that person super annoying for posing like that)

I mean it's not like forbidden, but if you aggressively race to scoop someone, then - depending on the details - people can reasonably be upset at you. Additionally, you're likely adding very little value by doing that compare to the counterfactual where you just prove some other results and don't stoke controversy.

My basic stance here is that I respect doing hard things as opposed to being cheap and doing things after you found out they're easy. This is of course not black and white. E.g. if you warn people ahead that you're going to go after their thing... (read more)

I think you're being dense on this. We can agree that furthering knowledge is good and shouldn't be gatekept, but this doesn't imply greedily racing towards results is decent behavior.

It can indeed be destructive to try to beat others towards a goal based on information that they are heading there. This disincentives openness and ambition for the less resource intensive parties, since they're much less likely to get appropriate credit for anything they try to pursue.

I find it super scary hearing you say this! Please don't get polarized by random disagreements about form.

I understand why one might be reluctant to use it, but demon summoning is still the right metaphor in so many ways.

The problem is that you'd have to figure out a technique that can *only* achieve the latter but not the former.

Arguably if you had a solution to alignment, it'd make human power concentration/coups worse. I don't personally have a strong opinion whether this is true, but seems plausible that this makes it bad (except for those in control of the AI)?

For a robust solution you'd need to coordinate your aligned AI development with an oversight mechanism that helps ensure the gains accrue to all. (Or at least ensure the people who gained power are in fact benevolent, set up a good system and only grab a moderate amount of resources).

Ah, the AI box experiment.

To me it was more that they both looked like "Basic points on AI safety" type posts (the "Why focus on extinction risk" post is not exactly that, but to the extent that it isn't I disagreed somewhat and didn't find it particularly insightful otherwise).

You might indeed have different norms and it depends a bit on the circumstances. If you say already knew how to do it (in the sense of having developed the idea somewhat) before talking to your colleague, and you just start working it out more to see if you're correct given the information from them it seems fine.

I do think it's in pretty bad taste if you didn't know beforehand you were able to do it, and you only tried hard after it became clear that it's probably kind of doable due to their work (and you know say that you're more overall capable/have mo... (read more)

They were broadly gesturing at taking the general norms of the math community seriously and signaling deference and respect. One specific norm I'm thinking of is of not racing towards something just because someone else's ongoing work has indicated a solution is within reach (admittedly not an absolute norm especially with certain very competitive people, but doing this is at least bad taste).

My main issue is that they were simply dishonest by playing up their respect for the community (which we now know was at best not deep-rooted). There was always goin... (read more)

I also rather they'd care about alignment, but I don't think upholding some norm gives you some moral clearance to break it in cases where you really want to - and if that happens you were probably not really caring about the norm so much!

E.g. one alternative would have been to straightforwardly tell mathematicians that they won't hold back publishing AI results, and understand that this may upset the community. This would've still upset the community, but in a more "honest" way.

I think you can reasonably not want to be associated with Elmore, without that being an issue of her integrity. (I haven't seen any good argument that her integrity is in question?)

I'm not sure what Coxon's reasoning was, but the effect on the public might be due to something like this:

If you're working on safety, it's already likely you're kind of concerned, and if you quit it's not clear whether that's because the situation is bad or because you're too scrupulous.

If you're working on capabilities, then instead you're most likely not the type to worry so much about the situation, so if you ring the alarm, it must really be pretty serious.

Note that this is kind of dumb, but I'd guess this models the situation pretty well, and I thin... (read more)

It's about who contributed significantly to the final solution in terms of developing the necessary mathematical ideas (and I believe it's unclear what to do if an AI did all the intellectual labor).

This is not something you can hand out. Clearly there's enough ambiguity so that different reasonable arrangements can benefit different people, so it's still possible to be more or less respectful when you go about creating outputs, but generosity is really the wrong frame. (In some cases, like if you're an advisor, and you invite your student to help on your work it could apply)

If the pressure tactics described by Buckmaster here are true, I think this is a really bad showing of OpenAI and Bubeck should reasonably face consequences. It doesn't seem like either party is super trustworthy though, so it's unclear whom to believe.

I don't think generous is the right word here. It's not like they did anything particularly impressive themselves. Bringing a bigger excavator to dig for the gold that everyone suspects is there is hardly an achievement.