Quick Takes

Here are the top posts LessWrong showed me when I looked at it today. This is the Recent display. It has tools to adjust what content you're seeing by up or downweighting tags. Here's mine. It's a pretty predictable set of interests.[1] I have AI set to 0.01, the lowest the system will allow me to move something. Currently I'm getting shown 1. A story about AI 2. A post about AI 3. The open thread 4. A post about an AI company 5. A post about AI 6. A post about doing math. It's about AI 7. A post about an AI company 8. A post about the AI safety field 9. A post about an AI company 10. A post about crypto, which is NOT about AI but isn't something that excites me. 11. A post about AI 12. A post about decision theory. The title is a direct reference to a previous AI post. 13. A post about agency! Oh, no, it's actually about AI agents. 14. A post about an AI company 15. A post about an AI paper Oh, plus the Best Of LessWrong banner is an AI post. ("Pay Risk Evaluators in Cash, Not Equity.") This isn't an unusual range of topics to see when I show up on LessWrong. I haven't been taking screenshots like the above at regular intervals, but it feels like this is a typical spread. Two years ago I said there was a lot of content about AI on LessWrong, I was mostly tuning it out with filtering tools[2] and I wanted some way to spend more time in the subsections I was interested in. I proposed some kind of subdomain setup to make the category boundaries stronger. I thought AI was smothering other topics. That problem has become worse. Today I swapped the AI tag to Hidden. That didn't work. I still see the curated posts ("I Did This Without Your Knowledge" and "GLM-5.3 Sometimes Considers How Train-Time Actions Shape Future Behavior") and I still see "Some Memories Of Earth Anthropic" and "Subtly Wrong Advice Misaligns Models More Than Obviously Wrong Advice" (which I guess aren't tagged at all?) I don't know what to do to make LessWrong as a w

I've found when studying steering vectors that the J-lens restricted to emojis often gives me the best hypotheses on what a direction is doing. For instance, here is the Assistant - Base model averaged over a few identical pieces of text (using the base model J-lens) + Assistant-Base: "🎓", " 😊", " ✅", " ❤️", "📚", " ✨", " 🥰", " ⭐", "🤝", "🔍", "🧭", "🌐", "🎙", "💡", "🔬", " 🤗", "🧘" - Assistant-Base: "😬", "😒", "👿", "💩", "😳", "😹", "😜", "😛", "😰", "🙄", "😠", "😂", "😨", "🚬", "🤑", "😏", "😝", "🙈", "😑", "👹" As another example, the word readout seems to suggest Arabic vs Sea, + Arabic, 阿拉伯 (Arab/Arabic), Arabic, árabe (Arab/Arabic), 沙漠 (desert), Saudi, arab, arabic, arabe (Arab/Arabic), Arab, arabic, arab, Saudi, Riyadh, Islamic, Arabian - marine, seaweed, fisherman, Fisheries, maritime, fishermen, Newfoundland, Maritime, Vancouver, Baltic But actually this is a desert/arid versus sea direction, and the names are only places associated with the desert. The emoji readout makes this more clear + 🏜 🐪 🐎 🐫 🕌 🏇 🕋 🌵 - 🐟 ⚓ ⛴ 🚢 🐳 🐋 🐠 🛳

Today's LLM agents are so smart, in so many ways... and yet they constantly make fairly basic/dumb mistakes, in my experience. This has been my experience with coding agents since day one, and does not seem to be improving. At the start of this year, they weren't good enough to be of much use to me; now, they're useful on net because they're so fast and because they know a bunch of obscure stuff, but they still make constant blunders, just like their predecessors. Working with these things is often frustrating, but it's also a weird sort of ego boost. I'm starting to feel like Zaphod Beeblebrox in the Total Perspective Vortex. People say "this new model is scarily capable, it's finding zero-days, it's solving longstanding math problems," and I say "wow, let me try it out," and then I do, and right away I see it doing a bunch of objectively suboptimal things and making a bunch of confused or simply false statements. And it's like: wow, I must be pretty smart, huh? This thing's a genius, sure -- but it's no match for little old me! And yeah, yeah, I know that human intelligence and model intelligence work differently, I know "capabilities are spiky," etc. Still, though, it is becoming increasingly unclear what co-working process I should expect to work well. If I keep the agents on tight leashes and stay continually available -- "in the loop" at every granular step of that loop -- this works well because it gives me a chance to push back on the mistakes, but this feels wrong somehow because the agents are being trained for more and more autonomy over longer and longer horizons. But they just... do not have the judgment needed to do nontrivial things in that setting? They're like smart-but-naive first-year grad students: book-smart, hard-working, lacking in taste/discernment and ability to prioritize, liable to go off on silly low-value tangents (which they treat with the utmost seriousness) if not closely watched and steered. But I can't just say "okay, they're

The US are the Big Scary I don't really know how to say it otherwise, and I don't have the mental health to turn this into a full post. The US are the biggest actor in AI. Their companies are firmly nationalist, using arguments such as "we need to have the best AI to defeat our autocratic opponents". The US have whimsical politics which can decide to ruin a foreign country; Trump, POTUS now and for the critical coming years of AI takeoff, does not have the interests of even nominal allies at heart. I often see arguments on LW about China, but they always land flat for me because China is looking very responsible on AI safety compared to the US! Even if the US don't destroy the world out of misalignment, they'll probably vassalize it out of power concentration. I just don't see how my non-US, democratic, liberal, non-AI-pilled country can avoid becoming a third world state dependent on the US for AI goods and services, even if it doesn't end up outright conquered. Please don't assume that we want the US to win.

https://borretti.me/article/we-wont-just-hang-out :(

MATS Residency applications close this Friday, Oct 31. The Residency is built for experienced researchers with an established track record, ready to do independent, full-time AI safety work. Residents get: • $155k–$285k compensation • Abundant compute • Support from leading AI safety researchers If you know someone with a strong research background who's been thinking about AI safety, please send this their way. Apply: matsprogram.org/residency

AI safety research is incredible for my life satisfaction and overall happiness. I expected that at some point it would peter out, but I just keep getting happier and more content; e.g., this is the best month of my life, but so was the last, and the last, and the last (up until April 2024). I feel like I contribute to something important, do technically complex and intellectually satisfying work, and can support myself and people I care about. I was worried that it would be some type of dysfunction (like a manic attack), but I feel pretty grounded, still get discouraged/unhappy at times. And I feel I have friends and coworkers who I trust who would tell me if that was the case (both professionally, leaning too much in the slop direction, and personally, if I was acting very weird; possibly this is also a chance for people I know to tell me that they think I'm cooked).