I think you could approximately define philosophy as "the set of problems that are left over after you take all the problems that can be formally studied using known methods and put them into their own fields." Once a problem becomes well-understood, it ceases to be considered philosophy. For example, logic, physics, and (more recently) neuroscience used to be philosophy, but now they're not, because we know how to formally study them.

So I believe Wei Dai is right that philosophy is exceptionally difficult—and this is true almost by definition, because if we know how to make progress on a problem, then we don't call it "philosophy".

For example, I don't think it makes sense to say that philosophy of science is a type of science, because it exists outside of science. Philosophy of science is about laying the foundations of science, and you can't do that using science itself.

I think the most important philosophical problems with respect to AI are ethics and metaethics because those are essential for deciding what an ASI should do, but I don't think we have a good enough understanding of ethics/metaethics to know how to get meaningful work on them out of AI assistants.

What does economics-as-moral-foundation mean?

He mainly used analogies from IABED. Off the top of my head I recall him talking about

• predicting where the molecules go when you heat an ice cube
• LLMs are grown; the companies aren't building crops, they are building farm equipment that grows crops (I don't remember this one from IABED)
• we know ASI could build a self-replicating solar-powered factory because those already exist (i.e. grass)
• leaded gasoline as a case study of scientists/companies making something bad and being in denial about it, even to their own detriment
• many people thought nuclear war was inevitable, but it didn't happen, largely because the people in charge would be personally harmed by it

I'm talking about my perception of the standards for a quick take vs. post. I don't know if my perception is accurate

My perception is that it's not exactly about goodness, it's more like, a post must conform to certain standards*. In the same way that a scientific paper must meet certain standards to get published in a peer-reviewed journal, but a non-publishable paper could still present novel and valuable scientific findings.

*and, even though I've been reading LW since 2012, I'm still not clear on what those standards are or how to meet them

On a meta level, I think this post is a paragon of how to reason about the cost-effectiveness of a highly uncertain decision, and I would love to see more posts like this one.

I don't know Bores personally. I looked through some of his communications and social media, most of it seemed reasonable (I noticed his Twitter has an unusually small amount of mud-slinging). I did see one thread with some troubling comments:

This bill [SB 53] recognizes that in order to win the AI race, our AI needs to be both safe and trustworthy.

In this case, pro-safety is the pro-innovation position.

[...]

As a New Yorker, I have to point out that SB53 includes a cloud compute cluster & @GavinNewsom said in his signing memo "The future happens [in CA] first"

...but @KathyHochul established EmpireAI in April 2024. So, thanks to our Gov's vision, the future actually happens in NY 😉

Why I find this troubling:

• Bores seems to want to race to build AI. Racing shortens timelines and decreases safety.
• "pro-safety is the pro-innovation position" seems false? If AI companies maximize profit by being safe, then they'd do it without regulation, so why would we need regulation? If they don't maximize profit by being safe, then pro-safety is not (maximally) pro-innovation.
• I think our best hope for survival is that governments become sufficiently aware of the danger of AI that they agree to ban frontier AI development until we can figure out how to make it safe. If Bores is indeed pro-innovation on AI, then he would presumably oppose such a ban. My guess is the average Democrat would be basically fine with banning frontier AI if the political winds shifted that way, but Bores would have a more strongly-held stance, in which case he would be worse than the average Democrat (but still probably better than the average Republican).
• He calls out New York directing funding to a new AI research lab as if that's a good thing, which I don't think it is. (I don't actually know what EmpireAI is doing, I looked at their website but it doesn't really say anything, it says they only fund "responsible" research, but I really don't trust them to know what qualifies as responsible.)

Politicans are often pressured to say those sorts of things, so perhaps he would still support an AI pause if it became politically feasible. So these comments aren't overwhelmingly troubling. But they're troubling.

If those quotes accurately reflect his stance on AI innovation and arms races, then he might still be better than the average Democrat if the increased chance of getting weak-to-moderate AI safety regulations outweighs the decreased chance of getting strong regulations, but it's unclear to me.

I will note that this was the only worrying comment I saw from Bores, although I didn't find many comments on AI safety.

I think the strongest case for AI stocks being overpriced is to ignore any specific facts about how AI works and take the outside view on historical market behavior. I don't see a good argument being made in the quotes above so I will try to make a version of it.

I'm going based on memory instead of looking up sources, I'm pretty sure I'm wrong about the exact details of the claims below but I believe they're approximately true.

• The Mag 7 have a P/E of 32. Historically, when companies had a P/E of 32, their average future returns were much worse than average (my guess would be 0–3%).
• A study looking at the performance of the #1 market cap company found that the top company almost always went on to underperform, which is an argument against buying Nvidia in particular.
• Nvidia as a % of the total world market cap is currently the largest any company has ever been in history. When things go outside the normal historical range, that generally suggests the price is unsustainable.
• Historically, the market has systematically over-extrapolated earnings/revenue growth. Companies with excellent earnings growth for years 1–3 tend to have merely above-average earnings growth for years 4–6, but they're usually priced as if they're going to continue to have excellent earnings growth. Although that's just an average, and companies with very high market expectations still exceed expectations something like 40% of the time.
• Stocks and industries tend to exhibit 3–5 year mean reversion, i.e. stocks that perform particularly well for 3–5 years on average underperform the market over the following year.

(These are five different perspectives on the same general market phenomenon, so they're not really five independent pieces of evidence.)

On the outside view, I think there's pretty good reason to believe that AI stocks are overpriced. However, on the inside view, the market sort of still doesn't seem like it appreciates how big a deal AGI could be. So on balance I'm pretty uncertain.

Importantly, AFAICT some Horizon fellows are actively working against x-risk (pulling the rope backwards, not sideways). So Horizon's sign of impact is unclear to me. For a lot of people, "tech policy going well" means "regulations that don't impede tech companies' growth".

As in, Horizon fellows / people who work at Horizon?