but I think ideally these should have been done by a MATS scholar, or ideally by an eager beginner on a career transitioning grant who wants to demonstrate their abilities so they can get into MATS later.
A problem here is that, I believe, this is on the face of it not quite aligned with MATS scholars' career incentives, as replicating existing research does not feel like projects that would really advance their prospects of getting hired. At least when I was involved in hiring, I would not have counted this as strong evidence or training for strong research skills (sorry for being part of the problem). On the other hand, it is totally plausible to incorporate replication of existing research as part of a larger research program investigating related issues (i.e. Ryan's experiment about time horizon without COTs could fit well within a larger work investigating time horizons in general).
This may look different for the "eager beginners", or something like the AI safety camp could be a good venue for pure replications.
On closer inspection, I believe this does not add much towards understanding the described people's psychology.
Although the described reactions seem accurate, the analogy seems week and the posts jumps too quickly towards unflattering conclusions about the outgroup. In particular, the case of being forcibly moved by a company towards another location is an extremely radical action given our current social norms and thus people can be expected to be indignant.
On the other hand, organizations imposing large but longer term changes on societies without asking is the norm. such as introducing social media or the internet.
I don't quite think so? My impression is the criticism is that LW is too much of an echochamber, in that people just express agreement with each other too much but probably that is mostly not because of people being nice but folks just outright having very similar believes
I worry there is a bit of wishful thinking involved in the high number of upvotes. I struggle to usefully pinpoint where my vibes disagree, but my impression is that the world is rife with examples of the powerful behaving badly and totally getting away with it.
If I look at corporate scandals, the case for fraud and nefariousness seems pretty strong. Just off the top of my head: Dieselgate where Volkswagen deliberately deceived official testers about the level of pollution their cards produce, Bayer knowingly selling AIDS laced blood for transfusions, Goldman Sachs selling off the assets they knew to be worthless just prior to the crash 2008. Sure, often fines are involved if caught, but the reputation of the companies remains surprisingly intact and overall if you take into account all the cases where they were not caught I'd doubt the crimes were not worth it financially.
On an individual level it's a bit murkier whether it's worth it. I guess for most people the stress of violating norms and being caught will cause a net wellbeing loss. That notwithstanding, given how much our society defaults to trust, there are lots of low hanging fruit for slightly motivated and competent ruthless people. One example only: outright fabricating large parts of your CV seems relatively rare despite the low likelihood and consequences of being caught.
But NYC is for ambitious conformists like bankers and lawyers. From that perspective there is no point in making a fortune in SF if your social circle does not appreciate your expensive Armani suit or your excellent taste in business cards. At least that's my vibes, I haven't been in over 15 years.
Narratively I'd pick Shenzhen. Scrappy little fishing village that transformed itself into a high tech hub seems suitable for building AGI.
I have heard Peter Thiel make the point that almost all the recent significant advances are concentrated in the digital world, whereas change in the analog world has been very marginal.
So:
- Not sure whether "weak government" applies. Factually, it does not seem like the federal administration is very interested in taking action on regulating AI, but on the other hand I do not think most voters perceive the Trump government as weak? I am also not sure I understand the point, but at any rate currently the moderates are not willing to denouncethe actions publicly very much?
- There is some momentum, but change does not seem to happen likely anyway. Arguably, after the FLI letter two years have passed without major changes.
- I don't think we are at peak activism yet, but also the movement is not clearly declining either.
- There is strong polarization according to some viewpoints: Twitter and certain parts of the administration are very happy to point to "radical doomers" for discreditation.
Overall, I think these are a wash, but no strong yes. At any rate, I have the hunch that these factors were derived from large social movements whose path to impact was very bottom up and around high saliency issues to the public. At least for the moderates in this debate this is not the path to impact I think, it's rather influencing key stakeholders.
1I just wanna state this gives me a feeling of the blind leading the blind.
I think I agree with this. To illustrate: When I met John (which was an overall pleasant interaction) I really did not think that the hat and sunglasses looked cool, but just assumed this was Berkeley style idiosyncracy. Usually I would not deem it appropriate to comment about this publicly, but since it was used as an example in the post this feels like a relevant enough data point to bring up.
1Thanks for publishing this!
My main disagreement is about a missing consideration: Shrinking time to get alignment right. Despite us finding out that frontier models are less misaligned by default than [1]most here would have predicted, the bigger problem to me is that we have made only barely progress about crossing the remaining alignment gap. As a concrete example: LLMs will in conversation display a great understanding and agreement with human values, but in agentic settings (Claude 4 system card examples of blackmail) act quite differently. More importantly on the research side: to my knowledge, there has neither been a recognized breakthrough nor generally recognized smooth progress towards actually getting values into LLMs.
Similarly, at least for me a top consideration that AFAICT is not in your list: the geopolitical move towards right-wing populism (particularly in the USA) seems to reduce the chances of sensible governance quite severely.
Less risk. AI is progressing fast, but there is still a huge amount of ground to cover. Median AGI timeline vibes seem to be moving backwards. This increases the chance of a substantial time for regulation while AI grows. It decreases the chance that AI will just be 50% of the economy before governance gets its shoes on.
This seems basically true to me if we are comparing against early 2025 vibes, but not against e.g. 2023 vibes ("I think vibes-wise I am a bit less worried about AI than I was a couple of years ago"). Hard to provide evidence for this, but I'd gesture at the relatively smooth progress between the release of ChatGPT and now, which I'd summarize as "AI is not hitting a wall, at the very most a little speedbump".
Less risk. AI revenue seems more spread.
This is an interesting angle, and feels important. The baseline prior should imo be: governing more entities with near 100% effectiveness is harder than governing fewer. While I agree that conditional on having lots of companies it is likelier that some governance structure exists, it seems that the primary question is whether we get a close to zero miss rate for "deploying dangerous AGI". And that seems much harder to do when you have 20 to 30 companies that are in a race dynamic, rather than 3. Having said that, I agree with your other point about AI infrastructure becoming really expensive and that the exact implications are poorly understood.
- ^
I think about two/thirds of this perceived effect are due to LLMs not having much goals at all rather than them having human compatible goals.
2
FWIW, Daniel Kokotajlo has commented in the past:
> If there was an org devoted to attempting to replicate important papers relevant to AI safety, I'd probably donate at least $100k to it this year, fwiw, and perhaps more on subsequent years depending on situation. Seems like an important institution to have. (This is not a promise ofc, I'd want to make sure the people knew what they were doing etc., but yeah)