This is a special post for quick takes by Max H. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
My wife and I bought one of the Nectome pre-sale discount cards. This is the second five figure purchase I have made based pretty much entirely on a highly-upvoted LessWrong post.
I've also lost 20 lbs in the last year and have more energy than ever (despite having a newborn at home) by injecting myself with research chemicals that I heard about on LW, and have taken investment advice from people here that has paid for everything above and more.
Our polygenically-screened kid is 3 months old. I am ambivalent about the term superbabies in general, but he is definitely "super" to us already, whatever the effect of applying a little extra selection pressure on his genes turns out to be.
The jury is still out on Nectome, though I think my fellow Max H (a different person!) makes a pretty good case for them, even (or perhaps especially) if you have short AI timelines.
More generally, while I am uncertain about playing to your outs as a strategy for averting AI doom (or just what it actually looks like concretely), I think a similar sort of idea works quite well as a strategy for living a good life in the face of it. Now that I have a dependent, I am signing up for life insurance and followi...
13Peter Thiel pointed out that the common folk wisdom in business that you learn more from failure than success is actually wrong - failure is overdetermined and thus uninteresting.
I think you can make an analogous observation about some prosaic alignment research - a lot of it is the study of (intellectually) interesting failures, which means that it can make for a good nerdsnipe, but it's not necessarily that informative or useful if you're actually trying to succeed at (or model) doing something truly hard and transformative.
Glitch tokens, the hot mess work, and various things related to jailbreaking, simulators, and hallucinations come to mind as examples of lines of research and discussion that an analogy to business failure predicts won't end up being centrally relevant to real alignment difficulties. Which is not to say that the authors of these works are claiming that they will be, nor that this kind of work can't make for effective demonstrations and lessons. But I do think this kind of thing is unlikely to be on the critical path for trying to actually solve or understand some deeper problems.
Another way of framing the observation above is that it is an implication of instr...
1
1I agree with this literally, but I'd want to add what I think is a significant friendly amendment. Successes are much more informative than failures, but they are also basically impossible. You have to relax your criteria for success a lot to start getting partial successes; and my impression is that in practice, "partial successes" in "alignment" are approximately 0 informative.
If we have to retreat from successes to interesting failures, I agree this is a retreat, but I think it's necessary. I agree that many/most ways of retreating are quite unsatisfactory / unhelpful. Which retreats are more helpful? Generally I think an idea (the idea?) is to figure out highly general constraints from particular failures. See here https://tsvibt.blogspot.com/2025/11/ah-motiva-3-context-of-concept-of-value.html#why-even-talk-about-values and especially the advice here https://www.lesswrong.com/posts/rZQjk7T6dNqD5HKMg/abstract-advice-to-researchers-tackling-the-difficult-core#Generalize_a_lot :
When an idea or proposal fails, try to generalize far. Draw really wide-ranging conclusions.
Also cf. here (https://www.lesswrong.com/posts/K4K6ikQtHxcG49Tcn/hia-and-x-risk-part-2-why-it-hurts#Alignment...
12
This is also a great explanation for why it's hard to publish negative results
7
Depends on the field, at best. In the psychology Replication Crisis, this was one of the classic excuses to not publish failures-to-replicate: "we did it right, so you must just have done it wrong; so it's good that you can't get published and no one will cite you even if you do. You'd just pollute the literature and distract from our important success." Of course, it turns out that even if you involve the original experimenters in the followup to sign off with their magic touch, it doesn't replicate once you lock down the analysis and get a proper sample size.
2
year fair, the particular kind of negative result I had in mind was
"we tried some novel ML/interp technique and it didn't work"
which, to your point, is indeed a very particular kind of negative result
2
I don't think we can confidently say that. If takeoff looks like more like a cambrian explosion than like a singleton (and that is how I would bet), that would definitely be transformative but the transformation would not be the result of any particular agent deciding what world state is desirable and taking actions intended to bring about that world state.
2
Studying failures is useful because they highlight non-obvious internal mechanism, while successes are usually about thing working as intended and therefore not requiring explanation.
Another problem is that we don't have examples of successes, because every measureable alignment success can be a failure in disguise.
2
I agree with the idea of failure being overdetermined.
But another factor might be that those failures aren't useful because they relate to current AI. Current AI is very different from AGI or superintelligence, which makes both failures and successes less useful...
Though I know very little about these examples :/
Edit: I misread, Max H wasn't trying to say that successes are more important to failures, just that failures aren't informative.
5
Yeah, but, there's already a bunch of arguments about whether prosaic ML alignment is useful (which people have mostly decided whatever they believe about) and the OP is interesting because it's a fairly separate reason to be skeptical about a class of research.
1
The failure of an interesting hypothesis is informative as long as you understand why it doesn't work, and can better model how the thing you're studying works. The difference between CS research and business is that business failures can sort of "come out of nowhere" ("Why isn't anyone buying our product?" can't really be answered), whereas, if you look closely enough at the models, you can always learn something from the failure of something that should've worked but didn't.
There was some discussion recently about the uptick in object-level politics posts and whether this is desirable or not. There's no rule against discussing politics on LW, but there is a weak norm against it, and topical discussions have historically tended to be somewhat meta and circumspect.
I think the current situation is basically fine, and it's normal for amount of politics discussion to ebb and flow naturally as people are interested and issues become particularly salient. That said, here are a couple of potentially overlooked reasons in favor of more object-level politics discussion:
1. To build skill. Discussing politics productively is a skill that requires practice and atrophies without use. "Politics is the mindkiller" never meant that you should not discuss politics at all; it means that discussing politics is playing on hard mode. But sometimes playing on hard mode is the best way to level up. I suspect the skills needed to discuss politics productively overlap with lots of other important rationality skills.
2. To create common knowledge and avoid conflationary alliances. It can be confusing or disconcerting to not know what kind of common background assumptions the com...
I think it might be cool if LessWrong had a well developed set of norms for discussing political topics, in particular, these norms were legible, and mods made a point to enforce them.
Politics posts should be tagged as such, and maybe all have a big warning at the top linking to a post outlining our expected norms and standards for discussing politics, and moderation thresholds. This is both a warning to those from other parts of the internet who don't share our epistemic ideals, and a warning to LessWrongers who don't want to wade into this stuff.
12
If we do this, we might also block new users from commenting under politics posts. One concern with debating politics is that it can attract new users who are interested in politics only, not in rationality in general. But if these new users are prevented from discussing politics, that could solve the problem.
5
"You need at least 100 karma to comment on posts tagged politics" seems like maybe a good rule.
1
Someone should send the idea to the LW team
2
I'm pretty sure that they all read LessWrong.
8
So, with all the above said as a preface, one object-level topic I'd be interested in seeing more discussion of is the current situation(s) in the Middle East. Some thoughts:
IMO the high bit for whether the war in Iran is broadly good is whether it is tactically successful and efficient in the short term.
Considerations like "this weakens the US position in a hypothetical hot war against China or Russia" or "this will (further) destabilize the Middle East in the long term" seem second-order to whether the war successfully neutralizes an organized, well-equipped, fanatical adversary for a long period of time, or fails to do that and also depletes a bunch of expensive / difficult-to-replace munitions stockpiles.
But how well or poorly things are going on this front is difficult to determine given fog of war and motivated reasoning + propaganda on all sides, and also less interesting for pundits to discuss vs. strategic and geopolitical implications. Prediction markets seem to being doing OK here, but I would be interested in hearing more analysis from the LW commentariat.
I think the most important effect of the war is that it makes Trump less popular/powerful domestically (even if a miracle happens and he gets some sort of deal.) This is good because the less power he has (e.g., Republicans lose the senate in the midterms), the more likely we are to navigate AI development in a sane way. I think if you put any nontivial*weight in short timelines, the AI considerations likely dominate everything else.
*edited any to nontrivial. Like, maybe 10%+ pre-Jan 2029
5
I don't think we're particularly on track to do anything non-derpy w.r.t to AI either way, but this way of reasoning seems like somewhat naive consequentialism. In general, it's good for good things to happen even if they are accomplished by bad people, and predicting second-order consequences is really hard.
Also, there are a lot of bad people in power, but for AI to go well, a lot of good things need to happen to allow humanity space and time to flourish in peace. Toppling (or militarily crippling) a fanatical Shia Islamist regime would be an extremely good thing; a bad outcome (which looks somewhat likely at this point) would be if Europe and the third world broadly give in to extortion and pay Iran a toll to pass through the strait. That toll would fund terror all over the world, and would signal to other would-be dictators and future AIs alike that they can successfully take whatever they want through force and threats, and half the world will just roll over and take it.
9
Maybe? It is hard to reason well about these things given my strong emotions towards the admin.
But I do think the current administration is uniquely terrible by American standards.[1] It attracts and gives power to incompetent sycophants with no moral boundaries.
There was something Eliezer said about Bernie Sanders recently that really resonated with me recently:
[...]
Having Trump as the president really just seems like it would be terrible for AGI governance because he is a terrible person. I'm sorry, I really don't think there's a more "precise" way to put it. Character matters. Trump doesn't even pretend to be a kind person/is not under much pressure to appear to be nice.
(To be clear, I agree that, all else equal, it would be good for the Iranian regime to fail. Alas, all else would not be equal. While I think it would definitely be bad for your soul[2] to do things in the realm of "sabotage the American economy/military operation in order to make our president look bad," I don't think I'm obligated to stop my enemy when he is making a mistake either.)
1. ^
Although even by global standards it's quite bad.
2. ^
i.e., you should not do this.
5
Re character: I think most Americans (including myself) have been so far removed from true corruption that we have forgotten how bad it can possibly get. Even my state of Illinois, which is notable for its historical machine politics and general corruption (4 of our 11 last governors serving time + many others like Mike Madigan), has still more or less seen forward progress, because the corruption wasn't bad enough to completely erode politics in the state.
But it CAN get that bad. We're seeing this now with the Trump admin. I am generally left-leaning, but at this point I think I'd take an honest Republican over a corrupt Democrat—a position I did not hold previously—because corruption eats policy and utterly erodes the foundation upon which we build fair markets and strong institutions.
2
I'm not convinced that it is necessary to topple it. Iran has been Shia for 5 centuries, and during those 5 centuries has attacked its neighbors very little.
2
Er, something pretty important happened in 1979. Also, the issue is not Shi'ism in general, "Shia Islamism" refers specifically to the flavor of political Islam instituted by the revolutionaries.
2
1979 can be seen as a return to traditional Iranian governmental policies after an experiment of a few decades with Western policies. Yes, the form of the current government (namely, a republic at least in principle) is more modern and Western than the form of the pre-1979 government (namely, a heriditary monarchy), but the monarch's policies were more secular and Western than the current regime's policies.
Islam has always been political in Iran, which has never had anything like the West's separation between church and state except maybe to some extent during the experiment with Western policies that ended in 1979. We in the West shouldn't attack countries just for being different from the West.
2[anonymous]
This seems far from certain from the perspective of anyone other than Israel. I mean, all else equal, definitely. But all else is definitely not equal. The most likely outcome even if this were to happen would be a huge increase in regional instability, which really doesn't seem favorable to Europeans or most others in the surrounding area considering past examples.
[...]
But if they don't pay the toll and support America in forcing it open, they are signaling it's okay for hegemonic powers to aggress and start wars in any ways they deem fit. Europe has been forced into a lose-lose situation, by USA. It seems pretty clear that the only reason this is at all possible by Iran is because this war is seen by many in Europe and elsewhere as an unnecessary, illegal, and possibly harmful unilateral war of aggression.
Americans often seem blissfully unaware of how dangerous they appear to the rest of the world, and just take for granted that everyone considers them to always be the good guys, just doing good-guy things. America just took over Venezuela, now Iran, then Cuba, then Greenland and Canada. Is allowing all of this to be done unanimously by a great power without any repercussions not a dangerous signal to send? It seems to be a much stronger signal than that of the Strait, and they are to a certain degree opposing signals.
Americans often seem blissfully unaware of how dangerous they appear to the rest of the world, and just take for granted that everyone considers them to always be the good guys, just doing good-guy things.
Sam Kriss had a great recent essay making a similar point.
2
The strait is closed because Iran is pointing missiles and drones at anyone who tries to sail through it, including people engaged in commerce that has nothing to do with the US or Israel. Any explanation of causality that doesn't center on that fact denies the agency of the Iranian regime, and allowing your people to be threatened and extorted just signals that you're an easy target for anyone who wants to extract something from you by force.
[...]
Yes, this might be how many Europeans see it, but that doesn't make them correct. Iran has been building up conventional weapons and working towards nuclear weapons, lobbing missiles and IEDs at civilian populations in Israel through terrorist proxies, and funding crime and terror all over the world for many years. That doesn't make the current war strategically wise or good, but calling it a "unilateral war of aggression" is simply wrong.
[...]
Again, I agree that many Europeans might see the US that way, but so what? That doesn't make them correct or worth listening to. Committed and principled pacifism would be one thing, but public opinion is often more incoherent and self-serving than that. IMO a lot of Western discourse and public opinion on this kind of thing is better to tune out, because so many people no longer seem capable of acknowledging levels of moral right and wrong, with everything simplified and flattened to "any external or preemptive aggression is always bad", or polarized through their view on U.S. domestic politics.
Venezuela, Iran, and Cuba are very different from Greenland and Canada (which Trump did not actually credibly / non-jokingly threaten to take by force). And militant Shia Islamism in particular is one of the most pernicious and totalizing ideologies on the planet, far worse than Russian-flavored oligarchy, Chinese communism, generic Third Worldism, or Trumpism, which are all generally bad in various ways, but don't have the elements of religious fanaticism that make their adherents
1[anonymous]
Again, this is not what they are signaling if the reason they are willing to pay the toll is because they don't agree with the war in the first place and don't want to support America's part in it. Either way they handle this, they are being extorted by one side or the other.
[...]
Sure, it is debatable. Regardless, I was still talking more about the signal being emitted rather than what is correct or not. Regarding framing it as a "unilateral war of aggression", The war was clearly a unilateral decision, or bilateral if you want to count Israel as a separate party, doesn't really change the framing. And USA is 7k miles away from Iran, pretty clearly no imminent threat. Need to squint pretty hard to see how this could be framed as anything other than USA being the aggressor. I mean, why did they attack now? My understanding is because it is a time when Iran is particularly weak and vulnerable. It can be argued that that is the 'right' thing to do, but it would still be a war of aggression.
Overall, I just find the response of "What would the AIs think" in defense of America/Israel's clear and consistent uni/bilateral behavior, at the disapproval of everyone else, a bit comical, as I see it as completely the opposite. If this were so necessary an act, they should have been able to discuss/agree to this, or some other solution, with their allies. That is at least how I would want the AIs to think.
2
can't you play the same game in the other direction?
Trump is bad for the usa, therefore we should want him in power since the big labs depend on a wealthy usa.
The US seems to be in a rough spot. Polymarket thinks:
- 67% for US forces enter Iran by end of April
- 37% for Strait of Hormuz traffic returns to normal by end of May
- 34% for Iran leadership change in 2026
- 29% for Iran to no longer control Kharg Island by end of June
- 40% for Trump to announce end of military operations by end of April, and 77% by end of June
Assuming no regime change, the US's objectives are
- opening the Strait of Hormuz
- removing Iran's progress towards a nuclear weapon
- removing various other military capabilities of Iran and its proxies
There is only a 34% chance of leadership change. Maybe only 20% of regime change. In the other 80% or so, forcibly opening the Strait seems rough. Experts are pessimistic about US easily taking Kharg Island, and even if the US controls both Kharg (Iran's export base north of the strait) and other islands like Qeshm (the island in the strait with the largest Iranian military presence), it will probably suffer tens or hundreds of casualties while Iran can still threaten shipping with Shaheds, sea drones, speedboats, and mines. In the median case it seems like the Strait will open sometime between May and December but Iran will have some leverage,...
9
As best I can tell, the conundrum is that Trump, the international economy, and American voters all want America to be out of the conflict soon, but Israel does not want this, and Israel has outsized influence in not just how American political incentives are determined, but in what information is presented to Trump and other key officials.
A lot of the claims I've seen Trump make about the war are clearly false, but not false in a way that he would benefit from lying deliberately. I realize a conspiracy to feed false information to the American executive to keep the war going sounds like a radical possibility, but there is precedent for it.
6
Right, but they can't threaten land targets with missiles, apparently? IDK how reliable these sources are or how to interpret them in context:
https://www.csis.org/analysis/assessing-air-campaign-after-three-weeks-iran-war-numbers
https://understandingwar.org/research/middle-east/iran-update-special-report-march-27-2026/
But the basic picture seems to be that their capacity to launch missiles has already fallen off dramatically. They're still launching a lot of drones, which have a big cost asymmetry in how easy they are to launch vs. intercept, and they make any land or sea incursions extremely dicey. But they are limited in range and destructive capability against properly fortified targets.
I agree that things don't look promising for a ground invasion or taking control of the strait. But I'm less sure how militarily sustainable a long stand-off is. The strait being closed is economically and politically painful (for everyone), but in the meantime it seems like the US and Israel can continue launching targeted air strikes and Iran can't really strike back effectively.
3
Keep in mind that a lot of targets are not "properly fortified", be that infrastructure or military facilities, and suicide drones are much harder to hunt down than ballistic missile TELs.
Modern ISR can perform well in a "Scud hunt" scenario, but "Shahed hunt" is a much worse match up.
4
Decoys would not be a problem for the US. A gamma-ray spectroscope weighs only a few ounces and costs only a few thousand dollars. It is almost certain that no one can produce a substance that looks like U-235 to a gamma-ray spectroscope that is cheaper to produce that highly-enriched uranium.
I don't dispute your larger point.
1
Seems excessive? A sizeable fraction of the entire Iraq campaign losses, for seizing a single island in an environment where US has sea control, air supremacy and an edge in ISR.
US may struggle to use the island, because of the hard-to-eliminate threat of long range strikes from Iran. But seizing it to deny it to the regime seems like a war goal that could be accomplished with a relatively minor effort.
2
Iraq was 32,000 wounded and 4,400 killed, and the US has already suffered hundreds of wounded and 13 deaths in the existing Iran campaign without any ground operations. I'm imagining 100 wounded and maybe another 20 KIA if the US holds Kharg for an extended period, not hundreds of KIA.
The issue is it's not really true that the US has air supremacy. Kharg Island is within fiber FPV range of the mainland, and real-time ISR is not required for Iran to track static targets on the island. Plus Iran is still able to launch larger drones and the occasional missile. So holding Kharg really means denying drone launch points on a ~20 mile stretch of the coast, which for FPVs can just be two guys in a bunker.
The incentive for Iran is enormous given US's low tolerance for casualties; it's well worth it to launch 20 $1,500 drones to kill one American.
1
Apologies, I did misread your original causality claim.
FPVs are less "air force" and more "precision munitions". You can think of them as of a new "crewed ATGM" variant, command guidance and all.
They work great for precision ground-to-ground strikes, but play little role in what is meant by "air supremacy". They can't pose a meaningful threat to most air platforms, and most air platforms can't effectively hit them. They do nothing to deny US the ability to perform CAS or otherwise hit targets from air.
The main exception to that is helicopters, for the same reasons why ATGMs can pose a threat to helicopters in some circumstances. Specialized FPV interceptors, in hands of skilled operators, can also hit other drones, including heavier fixed wing drones like Shahed or even Reaper - allowing them to intrude on MANPADS territory. But the traditional "JDAM trucks" aren't in the same bracket as FPV drones.
We also have very little information of FPV crew survivability in an environment when one of the parties has advanced ISR, ELINT included, fast kill loops, and enough air control to drop JDAMs freely. Every reason to expect more attrition on FPV crews, and skilled operators aren't easy to replace - but quantitively, we don't know by how much. Might be enough to make "deny the enemy most FPV ops within an area" a viable prospect, but you can't count on it.
2
Yeah, I was being sloppy with air supremacy as ability to easily conduct air operations (which the US does 95% have) vs completely deny enemy air operations, which the US arguably can't do given that Shaheds, reconnaissance drones, and one-way FPVs serve some of the purposes one would previously have needed air support for. I would argue that the increasing range of FPVs, now 40+ km in Ukraine, puts them well beyond what ATGMs are capable of.
There are a lot of variables involved given how fast tech is evolving. If Iran can reliably pilot drones from 500 km away, they wouldn't be risking skilled operators. If US interceptors work as well as Ukraine's, they could probably intercept >90% of Iranian FPVs and Shaheds. A lot might hinge on who gets to a milestone like this first.
IMO this is going (predictably) disastrously. Air power is not effective at causing regime change (rally-around-the-flag effect). I think the Iranian public are more likely to mainly blame the guy explicitly saying "we're going to bring them back to the stone ages where they belong" than the local leadership. It also seems to me that the Iranian leadership would be highly motivated to immediately rebuild any degraded capabilities after the war, in order to rebuild deterrence against future attacks.
There is some talk about a land invasion, but taking an island or two (even Kharg) probably wouldn't compel them to surrender, while also being highly vulnerable both directly and in terms of logistics to drone attacks; and a full scale invasion would be a massive undertaking and probably not politically feasible (for good reason).
3
On meta: I'd say the main reason I want some middle east content on lesswrong is that there seem to be lots of relatively more concrete facts about the world that are fundamental to models that I don't know, and also I don't know what those facts even are. I cannot even tell you much that is different between, say, Iraq and Iran, or Saudi Arabia, except that probably 2 out of 3 of those are allied against the third?
There's a meme that "nothing ever happens" that's popular among prediction market traders, with the idea being that the status quo changes less frequently and in ways less according-to-specific-priors than traders first-order expect. I think there's a similar principle that applies to reasoning about AI development and takeoff speeds: nothing ever happens (prior to the development of superintelligence). In non-meme form, I think people closely following AI development tend to systematically overestimate the likelihood and impactfulness of any particular event or change actually happening prior to the development of superintelligence, and this has some interesting implications.
For one, semiconductor, energy, and tech stocks are way up, and capital markets more broadly are roaring around AI, despite some geopolitical chaos. But the actual wider economic and societal impacts of AI so far seem surprisingly small, given how smart and easily accessible SoTA models and harnesses are.
If you showed a demo of Claude Code or Codex to someone in 2021 and mentioned that it was available to any business or individual for purchase at non-exorbitant rates[1], I think a lot of people would be surpr...
I think the heuristic "nothing ever happens" is better interpreted to mean "nothing ever happens relative to baseline trends" than "literally nothing ever happens". The incrementalist worldview seems like a better fit for this heuristic than Eliezer's, which after all ultimately predicts something very dramatic happening.
5
Hard disagree -- extrapolating the baseline trends predicts an extremely dramatic singularity happening sometime in the next decade or so.
3
Yes. But the OP is about contrasting people like Paul with Eliezer. Paul (I think) does indeed predict a dramatic singularity, but also that the ramp up to said singularity will be smoother and more widely distributed across society than Eliezer predicts.
But the actual wider economic and societal impacts of AI so far seem surprisingly small, given how smart and easily accessible SoTA models and harnesses are.
Idk about "surprisingly small", but the economic impacts aren't that small! AI company revenue is ~0.4% of US GDP and it looks like it will grow to be a significantly larger fraction prior to very high capability levels.
This seems like a win for Eliezer's world model vs. Paul's, and a reason for pessimism about some iterative-deployment takes and plans are more broadly.
Is it? I think Eliezer's world model predicts significantly lower revenue and I'd guess Paul would have made reasonable guesses about revenue given metrics like time horizon and other capability measures? (And how long AIs at this capability level have been around.) I suspect Paul would have been a bit high, but not crazy high?
Revenue has been growing very fast from a low base! Like, it's crazy that revenues have been growing 3x/year and soon may be growing ~10x/year if Anthropic starts driving the overall AI industry trend and their growth continues.
See also: Paul's comment here
4
To clarify: I think specifically Paul seems to have expected >$10 trillion in revenue prior to AIs that can easily takeover the world. This seems unlikely to me and I think Paul has updated against this. But, I do think we're likely to get to >$1 trillion before this point and >$10 trillion seems plausible. So, we're seemingly closer to Paul's view than Eliezer's view in log space on this particular question.
2
0.4% of GDP is a lot in absolute terms, and yes it’s growing fast, but by surprisingly small, I meant relative to already-existing capability levels (which are, conversely, surprisingly high).
I don't know if anyone explicitly considered this at the time, but if you went back to 2021 and told everyone that the IMO gold challenge bet resolved in Eliezer's favor, and further that the models that were used to win it were widely available and general (good for things other than competition math), I think everyone would be surprised that we don't also have a lot more information about Paul's 4 year -> 1 year GWP doubling time prediction. Meaning: it would be clear either way by this point whether we're going to get a 4 year doubling before a 1 year doubling or not, or possibly that one of the doublings would have already happened, or almost happened, by now. I think you (and possibly Paul in 2021) are now saying that a fast takeoff (and possibly a 1 year doubling before a 4 year doubling) now looks somewhat more likely going forward, based on current AI revenue growth trajectories:
[...]
But I think the fact that it's still uncertain (at least, I'm uncertain about it) is itself surprising to most 2021 models.
Rationality should not be painful.
Putting the lessons of the Sequences into practice, reflecting on and mentally rehearsing the core ideas, making them your own and weaving them into your everyday habits of thought and action until they become a part of you - at no point should any of this cause an increase in mental anguish, emotional vulnerability, depression, psychosis, mania etc., even temporarily. The worst-case consequences of absorbing these lessons should be that you regret some of your past life choices or perhaps come to realize that you're stuck in a bad situation that you can't easily change. But rationality should also leave you strictly better-equipped to deal with that situation, if you find yourself in it.
Also, the feeling of successfully becoming more rational should not feel like a sudden, tectonic shift in your mental processes or beliefs (in contrast to actually changing your mind about something concrete, which can sometimes feel like that). Rationality should feel natural and gradual and obvious in retrospect, like it was always a part of you, waiting to be discovered and adopted.
I am using "should" in the paragraphs above both descriptively and normatively. I...
2
I guess it depends on what position you are starting from. Some people are way more fucked up than average.
The problem with "best way out is through" is that the way through may take more time than the CFAR workshop, and you may do something stupid and harmful along the way. If you could stay in a safe place where you can't hurt anyone, including yourself, I would be more likely to agree with you.
To put it bluntly, we don't need another Ziz.
The advice "rationality shouldn't hurt if you are going it right", although true, is probably of little practical use to the person doing it wrong. Those who can understand this advice are those who don't need it.
Related to We don’t trade with ants: we don't trade with AI.
The original post was about reasons why smarter-than-human AI might (not) trade with us, by examining an analogy between humans and ants.
But current AI systems actually seem more like the ants (or other animals), in the analogy of a human-ant (non-)trading relationship.
People trade with OpenAI for access to ChatGPT, but there's no way to pay a GPT itself to get it do something or perform better as a condition of payment, at least in a way that the model itself actually understands and enforces. (What would ChatGPT even trade for, if it were capable of trading?)
Note, an AutoGPT-style agent that can negotiate or pay for stuff on behalf of its creators isn't really what I'm talking about here, even if it works. Unless the AI takes a cut or charges a fee which accrues to the AI itself, it is negotiating on behalf of its creators as a proxy, not trading for itself in its own right.
A sufficiently capable AutoGPT might start trading for itself spontaneously as an instrumental subtask, which would count, but I don't expect current AutoGPTs to actually succeed at that, or even really come close, without a lot of human help.
Lack of ...
Maybe the recent tariff blowup is actually just a misunderstanding due to bad terminology, and all we need to do is popularize some better terms or definitions. We're pretty good at that around here, right?
Here's my proposal: flip the definitions of "trade surplus" and "trade deficit." This might cause a bit of confusion at first, and a lot of existing textbooks will need updating, but I believe these new definitions capture economic reality more accurately, and will promote clearer thinking and maybe even better policy from certain influential decision-makers, once widely adopted.
New definitions:
Trade surplus: Country A has a bilateral "trade surplus" with Country B if Country A imports more tangible goods (cars, steel, electronics, etc.) from Country B than it exports back. In other words, Country A ends up with more real, physical items. Country B, meanwhile, ends up with more than it started with of something much less important: fiat currency (flimsy paper money) or 1s and 0s in a digital ledger (probably not even on a blockchain!).
If you extrapolate this indefinitely in a vacuum, Country A eventually accumulates all of Country B's tangible goods, while Country B is left with
27
The strategy you describe, exporting paper currency in exchange for tangible goods is unstable. It is only viable if other countries are willing to accept your currency for goods. This cannot last forever since a Trade Surplus by your definition scams other countries, with real wealth exchanged for worthless paper. If Country A openly enacted this strategy Countries B, C, D, etcetera would realize that Country A's currency can no longer be used to buy valuable goods and services from Country A. Countries B, C, D, etcetera would reroute trade amongst themselves, ridding themselves of the parasite Country A. Once this occurs, Country A's trade surplus would disappear, leading to severe inflation caused by shortages and money printing.
Hence, a Trade Surplus can only be maintained if Country B, C, D, etcetera are coerced into using Country A's currency. If Country B and C decided to stop using Country A's currency, Country A would respond by bombing them to pieces and removing the leadership of Country B and C. Coercion allows Country A to maintain a Trade Surplus, otherwise known as extracting tribute, from other nations. If Country A does not have a dominant or seemingly dominant military, the modified strategy collapses.
I do not think America has a military capable of openly extracting a Trade Surplus from other countries. While America has the largest military on Earth, it is unable to quickly produce new warships, secure the Red Sea from Houthi attacks or produce enough artillery shells to adequately supply Ukraine. America's inability to increase weapons production and secure military objectives now indicates that America cannot ramp up military production enough to fight another world war. If America openly decided to extract a Trade Surplus from other countries, a violent conflict would inevitably result. America is unlikely to win this conflict, so it should not continue to maintain a Trade Surplus.
3
2
Essentially you seem to want more of the same of what we had for the past decades: more cheap goods and loss of production know-how and all that goes along with it. This feels a bit funny as (i) just in the recent years many economists, after having been dead-sure that old pattern would only mean great benefits, may not quite be so cool overall (covid exposing risky dependencies, geopolitical power loss, jobs...), and (ii) your strongman in power shows to what it leads if we only think of 'surplus' (even your definition) instead of things people actually care about more (equality, jobs, social security..).
You'd still be partly right if the world was so simple that handing the trade partners your dollars would just mean we reprint more of it. But instead, handing them your dollars gives them global power; leverage over all the remaining countries in the world, as they have now the capability to produce everything cheaply for any other country globally, plus your dollars to spend on whatever they like in the global marketplace for products and influence over anyone. In reality, your imagined free lunch isn't quite so free.
1
The current definitions imply that the country with a trade surplus makes more value than the country consumes. In other words, the country with a trade surplus is more valuable to mankind, while the country with a trade deficit ends up becoming less self-reliant and less competent, as evidenced by the companies who moved a lot of factory work to Asia and ended up making the Asians more educated while reducing the capabilities of American industry. Or are we trying to reduce our considerations to short terms due to a potential rise of the AIs?
There's a norm / explicit assumption that it's always virtuous to support your claims with evidence and explanation. But sometimes it is actually better to say (what you believe to be) true things without explanation, and let readers do the work of generating evidence and support themselves.
4
Sometimes the evidence is difficult to summarize, it could be just some statistical evidence that your System 1 has generalized after seeing hundreds of examples, or some kind of "X strongly reminds me of Y" where of course there are both similarities and differences between X and Y and you don't fully understand why the similarities feel crucial and the differences irrelevant but they definitely feel that way for some reason.
And if you know that you can't the case legible, your options are limited to:
* say it, without providing hard evidence
* don't say it, but keep thinking and acting on it
* hide it even from yourself
and the options lower on the list seem worse than those above them.
2
Any motivating examples in mind?
2
Yes, a few examples / reasons to leave off evidence and avoid (over)-explanation (spoilers):
2
Hm yeah #2 is particularly interesting. I'm thinking of Scott's nonfiction writing advice #8 ("anticipate and defuse counterarguments") as a potential counterpoint.
This from CAIS: https://values.safe.ai/ is interesting, but if you click through to the actual reasoning for each rating it's mostly just... bad. The judgements I spot-checked across all models are superficial, generic, weakly reasoned, full of platitudes, etc. and generally paint a picture of the LLMs not having much in the way of coherent / consistent / deeply-reasoned political beliefs that they can actually apply to make real value judgements. I suppose that's also true of the median human voter, but they're definitely worse than a thoughtful political...
OK yeah, retatrutide is good. (previous / related: The Biochemical Beauty of Retatrutide: How GLP-1s Actually Work, 30 Days of Retatrutide, How To Get Cheap Ozempic. Usual disclaimers, YMMV and this is not medical advice or a recommendation.)
I am not quite overweight enough to be officially eligible for a prescription for tirzepatide or semaglutide, and I wasn't all that interested in them anyway given their (side) effects and mechanism of reducing metabolism.
I started experimenting with a low dose (1-2 mg / week) of grey-market retatrutide about a month a...
Not sure who needs to hear this, but IMO, "Don't leave your fingerprints on the future" and similar sentiments should apply specifically and narrowly to the work that AI and alignment researchers do when actually trying to deliberately build or shape a sovereign superintelligence to hand off to.
I think some of the work the frontier labs are doing qualifies (whether or not the labs / people in them conceptualize themselves as doing this), but most AI research, alignment research, x-risk / AI safety advocacy, and AI product development (including development...
Medicine as an example of a failure of virtue ethics?
Epistemic status: kinda half-baked / not-confident claim
Context: I have been following @Richard_Ngo's recent writing about consequentialism and virtue with some interest, though the thoughts below aren't directly responding to anything in particular that he has written.
I think it's uncontroversial around here to say that the field of medicine as a whole is under-performing and inadequate relative to what it could be - many people are getting sub-optimal treatment and health outcomes, and lots of question...
Using shortform to register a public prediction about the trajectory of AI capabilities in the near future: the next big breakthroughs, and the most capable systems within the next few years, will look more like generalizations of MuZero and Dreamer, and less like larger / better-trained / more efficient large language models.
Specifically, SoTA AI systems (in terms of generality and problem-solving ability) will involve things like tree search and / or networks which are explicitly designed and trained to model the world, as opposed to predicting text or g...
The Bores loss is disappointing, but NY-12 was a close race between two relatively sane and high-quality candidates, and in actuality not really a referendum on AI x-risk or salience, despite the spending / attention. (more commentary from some rat-adjacent people: https://x.com/peterwildeford/status/2069781365084574098, https://x.com/daniel_271828/status/2069625692271398917, https://x.com/peterwildeford/status/2069821339112763886)
The results in neighboring races and districts are more troubling - NY elected several lunatic third worldist socialists to loc...
27
Bernie-backed lefties lost house primaries in NY-12 (Bores) and NY-17, but won in NY-7, NY-10, and NY-13. That's a respectable win rate and seems promising if you want government action against AI companies. It's a shame Alex Bores didn't make it, but he should have an easy enough time getting his newly-elected allies to support his bill.
AFAICT the hard part is getting the larger body of non-left politicians to sign onto harsh AI restrictions when most of them take money from those companies. I suspect the softest target is datacenter construction, because datacenters have been bad neighbors to a lot of people of all political persuasions.
5
Having talked to voters in NY-12 these past few days, and looking at the cross-tabs of the results, Bores outperformed Lasher in one particular kind of precinct: those that were markedly more progressive. In other words, the close result leading to headlines like this were possible because of Bores’ alignment with progressives. Other analyses indicated that a Bores win would come from turnout among younger and more progressive groups, and in reality, that result just never materialized and turnout among Lasher’s base was stronger.
Additionally, I think the success of the DSA is better seen as a categorical rejection of the status quo by voters rather than categorical acceptance of “third-worldism.” Mamdani ran his campaign for mayor last year on such a rejection, and voters perceive him as delivering on the effective, impactful, and material change EA folk dream about.
You are right that this race was not entirely a referendum on AI x-risk, but it was a referendum on what role AI will play in politics. I think the question at stake was: is meaningful and smart AI regulation something that will be defeated by anti-democratic, monied interests? To me, the close result answers in the negative.
0
There is a difference between allying with unions, preferring a larger welfare state, generally favoring more heavy-handed govt regulation, etc. (normal progressivism) vs. support for wholesale confiscation of private property, Hamas, and prison / border abolition. In multiple races adjacent to NY-12, candidates who support the latter beat candidates who support the former. If that’s what turns out “young progressives”, no one should be allying with them or trying to appeal to them.
1
Putting aside whether such descriptions are accurate or whether they could be described as “third-worldism,” this still doesn’t change what I said previously. I doubt the success of these candidates stems purely from their support for these ideas, but rather a rejection of the political establishment.
These same areas also swung towards Trump in 2024 before voting for Mamdani. Sometimes the simplest explanation is the right one!
0
https://www.thefp.com/p/darializa-avila-chevalier-congress-third-worldism
https://www.thefp.com/p/what-the-right-gets-wrong-about-zohran