Quick Takes

Pinker's letter here seems pretty disingenuous to me. For instance, the arguments he mentions have been refuted to hell and back. Its fine to disagree with the refutations, but to repeat them without the corresponding refutations and why he doesn't agree I think speaks to a lack of seriousness. On the charge he's engaging in poor faith and with bullying, I think he's basically arguing that technically he's not, because he never called people names, only hypothetical constructs, thought experiments, and argumentative conclusions. This is pretty silly. Here is the first example he offers: [...] I think it is pretty self-evident that this particular defense is quite weak, but to state it plainly: In this world, if you say that someone is projecting "alpha-male psychology" that is indeed as good as a claim that that person is a red-pill incel edgelord and is therefore low-status compared to us high status people who are not red-pill incel edgelords. Thus such red-pill incel edgelords should not be taken seriously. Indeed, this is basically what Pinker claims above! Note that calling someone an "alpha male" is not actually an insult. If anything, it'd be a compliment. It is the accusation that one would even think in terms of "alpha/beta males" to come to their conclusions which is the insult. And that is exactly what Pinker claims he's doing! I think there are similar twisty turns of phrase throughout the article, where Pinker consistently says something technically true, but with connotations pointing in the opposite direction of the truth claim. The clearest example here is this quote [...] If you think extinction is indeed a real threat, then these sound like fairly reasonable reactions! You should be more skeptical of the "institutions of modernity", invest less, and probably should have fewer children if you think that The End is Nigh. As for mental health, ideally people would not remain depressed or stressed for long periods of time due to the threat of e

Optimization processes like gradient descent or hill climbing are often visualized using a 3D landscape with hills and valleys and a path walking down (or up) that landscape, like this: This is nice to give a very basic intuition for what's going on, but this type of visualization is also misleading in many ways. To my annoyance, in most cases where I've encountered such visualizations, they came without caveats about the limitations of the "metaphor". So here's a (surely incomplete) list of some of the ways in which optimization can behave very differently from what the "walk down a landscape" metaphor suggests: * In many optimization problems, the "landscape" you navigate has way more dimensions than the 2 input dimensions such a landscape has, which has many unintuitive consequences. * For instance, local optima are extremely rare in higher dimensions relative to saddle points (because at any critical point all first derivatives are 0; but for it to be a local maximum or minimum rather than a saddle point, the curvature must have the same sign in every direction, which happens exponentially less often with added dimensions). * In low dimensions, hill climbing and gradient descent appear about similarly useful. But in higher dimensions, hill climbing quickly becomes infeasible (because random vectors tend to be ~orthogonal to the steepest direction, so most steps will be wasted), so gradient descent in high-dimensional spaces tends to be much more efficient (at least assuming the gradient is easy to compute). * The "landscapes" of optimization processes are often not static but in constant, possibly unpredictable motion. When a company is maximizing its revenue, it has to constantly adapt to competitors, regulators, seasonal effects, and more. * Indeed, it can happen that the actions the optimizer takes actively shape the landscape, often unpredictably, i.e. path dependencies. Imagine a person is trying to convince another person of something. They can'

Automating alignment research is alignment-complete People throw “automating alignment“ a little too often. I feel like they don‘t grasp how hard it actually is. In order for you to trust alignment solutions produced by AI, you need to 1. Know that this AI is capable enough to produce an actual solution instead of making up some slop 2. While being capable, this AI is sufficiently aligned to a) not pull a coup over you and b) know which values to align to Those are basically the requirements for the “hard” ASI alignment problem. In other words, the AI you start off with should already be aligned, so the strategy of automating alignment assumes the problem away. How do you make sure that AI made by your AI will make sure that the AI it will create (will make sure that the AI it will create etc…) will be aligned? That sounds to me like an extremely hard problem with a bunch of failure modes and very sensitive to initial conditions, and it’s called the Tiling problem in the field of Agent Foundations. There are very few people who work on Tiling right now, and I doubt that Jacob meant this type of research by his tweet. It’s not the only path we could choose to make smart AI safe, but those paths do not look like “automating alignment“. They mostly look like trying to make corrigible AIs which stay corrigible, or something like Co-Agency, or human intelligence amplification, or some other weird stuff we haven’t even thought of (and I am very interested in discussing those!)

Between Sep 2025 and Sep 2026 was likely the fastest year of total param scaling (for properly RLVRed models) that LLMs will ever see. In Sep 2025, Opus 4.5 and Gemini 3 Pro weren't yet out, GPT-5.0 was GPT-4o with RLVR (maybe 600B total params), Sonnet 4.5 was the latest thing. In Sep 2026, we have Fable 5 and Astra 6, probably 10T-20T total params, and the next OpenAI model might be targeting Rubin, in which case it could be 40T-60T total params. Opus 5.5 being surprisingly capable weakly suggests Anthropic might also have a post-Fable model getting ready for next year's hardware. Very approximately, the shape of scaling is that 2T total param models were possible in 2025, 20T in 2026, 200T in 2028, and 2,000T in 2032 (which isn't too expensive to serve). It took 1 year to 10x the params in 2025-2026, it'll take 2 years to do the same in 2026-2028, then 4 years to do so yet again in 2028-2032. The 1 year of 2025-2026 saw the transition from Sonnet to Opus to Fable. A similar amount of qualitative progress might happen in the 2 years ending in 2028, and then in the 4 years ending in 2032. Without an algorithmic breakthrough like continual learning in a strong sense, this could be literally what happens. But the results of 2026 suggest LLMs might themselves be capable of inventing continual learning soon, breaking their paradigm. Probably this happens by 2028, if it happens at all in the current regime. So in addition to the background risk of a paradigm-breaking innovation driven by human effort, there's an unusual concentration of risk around 2027-2028, when the amnesiac slow-learning LLMs become superhumanly smart and the difficulty of continual learning is tested for the first time against their uneven capabilities.

There are fun jobs outside research too. When I was looking for a job one and a half years ago, I was operating under the implicit assumption that research jobs are the interesting jobs, while other jobs might be useful but would be less enjoyable. When I joined the European AI Office, I expected not to enjoy the job, and I considered it some amount of personal sacrifice that was worth taking due to the importance of the job. I was very wrong about this. I enjoy my time working on regulatory enforcement in the government more than I enjoyed any of my previous roles (research at ARC, research at MATS, a finance internship in a researcher role, mathematical research).[1] In retrospect, it was pretty dumb to make the assumption that research is the most interesting type of work when I hadn't really tried any other type of work yet. But I see other people making the same assumption, and not wanting to leave their research roles out of fear that other work wouldn't be as intellectually fulfilling. I suspect they are often wrong, and they would actually enjoy other roles too. Once you are trying to accomplish something in the world, there are so many possible actions to take. So many strategies to form, scenarios to map out, choices to make. Meanwhile the world keeps throwing new situations at you, there is always a fire to put out, and a next most important problem to tackle. To me, it often feels like playing a very complex strategic board game: you have limited cards, but if you play them right, you can acquire more cards and unlock new options, while helping your fellow players who will help you in turn, in the great cooperative game of "making hopefully good things happen in the world". If you have only tried research before, but you think there is a non-research role where you might have more positive impact, I recommend that you take the plunge and come join the game! 1. ^ By the way, the AI Office is hiring. If you are an EU citizen, consider applyi

Problem: there's not enough people discussing (non-prosaic) alignment ideas. Solution: LessWrong should pay people to do it. Below I'll get more in depth about the problem and the solution(s). Alignment Newsletter Rohin Shah was running the Alignment Newsletter. It was good for many reasons: * Rohin was helping to explain and spread ideas. * Rohin contributed to the discussion. * Rohin was a person who knew stuff from different alignment subfields (having such people is good). But Rohin stopped and now nobody does what he did. That's bad! Even if he didn't stop, having multiple people doing the kind of things he did would be good. Review LW has a reviewer, Justis. He's pretty great, but has two flaws: * He's just one guy. * He's not very familiar with alignment stuff. That's bad (for a single non-expert to do all of the reviewing)! Hire more reviewers / increase Justis' pay to make him more knowledgeable about alingment. A similar criticism was made by Roman Leventov, about the review system basically not existing. LessWrong Even when you look at posts (with specific ideas/proposals) of high-status researchers, the engagement with those posts is often pretty "meh". Example 1, example 2, example 3 (the comments are mostly about topics adjacent to the actual topic of the post). This is worrying for people who aren't high-status. lemonhope made a somewhat similar observation. Talented People There are a bunch of people who can engage both with philosophical and mathematical, informal and formal topics. And who have status in the community - 2000+ karma and connections. They could help tremendously with giving feedback and disseminating ideas. But right now they have very little incentive to do it. Like, one such person I know isn't even very active in this community. Why are we sitting on talent? Would be good if we could motivate such people. Pay to Play I know one person who paid money to get feedback. I did the same myself. Would be go

OpenAI IPO delayed over investor fears of Zvi full 'delenda est'.