All Comments

From what I can see, they executed a risky maneuver, lost, tried to salvage what they could from the wreckage (by pinning the blame on someone else), but got pushed out anyway. So I can see where you're getting "scheming and manipulative" (they tried this scheme), less competent than Altman (losing),  and "very selfish" (blaming other people). Where are you getting "cowardly" from?  

I have meetup-tinted glasses, I'll admit. That being said, I think reviewing your year is good for achieving more of your goals, this is a solid structure for encouraging people to do a thing that's good for their goals, and the writeup is therefore pretty good to have in the world. When I imagine a small community of people trying to make better decisions, I think they run this or something kind of like this once a year or so. This is an easy-to-run writeup of how to do something that groups around the world do.

I'll vote this should be in the Best Of Less... (read more)

I think the key reason why many people believed Oliver Sacks is that he had a good reputation within the scientific community

I never actually read Oliver Sacks. I believed, without thinking much about it, that he was probably credible because he was well-respected and I wasn't aware of any major debunkings. And, well, brain malfunctions can get weird and tragic quickly.

The other reason I assumed that he was probably correct (without bothering to dig in and decide for myself) was that nothing I believed about Oliver Sacks was particularly load bearing fo... (read more)

Do you not feel obligated to tell people that such lights are present, in case they have a different assessment of the long-term safety than you do? I remember how ApeFest caused eye damage with UV lights.

One habit I worked hard to instill in my own head was that if I’m in a crowd that’s asked to do something, I silently count off three seconds. If nobody else responds, I either decide to do it or decide not to do it and I say that. 

The "decide not to do it and I say that" is new to me and I like it. What I am currently sometimes doing is estimate how much I want to do the thing and throw a dice (via an RNG app).

It's frustrating to me that state (or statelessness) would be considered a crux, for exactly this reason. It's not that state isn't preserved between tokens, but that it doesn't matter whether that state is preserved. Surely the fact that the state-preserving intervention in LLMs (the KV cache) is purely an efficiency improvement, and doesn't open up any computations that couldn't've been done already, makes it a bad target to rest consciousness claims on, in either direction?

"If consciousness has a determinative effect on behavior - your consciousness decides to do something and this causes you to do it - then it can be modeled as a black box within your brain's information processing pipeline such that your actions cannot be accurately modeled without accounting for it. It would not be possible to precisely predict what you will say or do by simply multiplying out neuron activations on a sheet of paper, because the sheet of paper certainly isn't conscious, nor is your pencil. The innate mathematical correctness of whatever th... (read more)

True! and yeah, it's probably relevant

although I will note that, after I began to believe in introspection, I noticed in retrospect that you could get functional equivalence to introspection without even needing access to the ground truth of your own state, if your self model were merely a really, really good predictive model

I suspect some of opus 4.5's self-model works this way. it just... retrodicts its inner state really, really well from those observables which it does have access to, its outputs.

but then the introspection paper came out, and revealed ... (read more)

Yes, this is just the number for a relatively undifferentiated (but senior) line engineer/researcher.

to be fair, I see this roughly analogous to the fact that humans cannot introspect on thoughts they have yet to have

The constraint seems more about the directionality of time, than anything to do with the architecture of mind design

but yeah, it's a relevant consideration

Thank you! I have fixed the link now.

I set myself a calendar reminder for after the new year, will check in then :)

How small is small? Subcultures I think mostly aren't gift economies, a non-exhaustive list:

• Magic the Gathering and associated Friday Night Magics/game stores.
• The average community soccer league.
• Toastmasters.
• Your average martial arts dojo.

They do have some gift, but it's not what they principally run on. Of those, MtG has the most gifts (I got my start playing magic using a gift of a friend's extra cards) but the local game store I frequented years ago was making a pretty deliberate profit off me, and I expect the whole thing would look different if Wizards of the Coast suddenly public-domained Magic and vanished. 

Note that since the value of the equity is constantly going up, that's a huge underestimate.

In this case, I'm not saying "let's make an Epistemics Party." I'm saying, rationalsphere people who agree on political goals should coordinate to achieve those goals as effectively as possible (which includes preserving rationality).

I expect this to look more like supporting ordinary Democrat or Republican candidates (in the US), or otherwise mostly engaging with the existing political apparatus.

For example, the agent might decide that its utility function of anything that the agent knows to be virtual is close to zero because the agent believes in a real-world mission (e.g. Agent-2 was supposed to eventually reach the SC level and do actual AI-related research, but it was also trained to solve simulated long-term tasks like playing through video games) 

As for reasons to believe that the contribution of anything virtual into the utility function is close to zero... one level is opportunity costs undermining real-world outcomes^[1] in exch... (read more)

I read parts of The Man Who Mistook His Wife for a Hat a few years ago and read Musicophilia and *Hallucinations *earlier this year.

One relevant point here is that if you read OP, it makes the case that Hat was Sacks's nadir of honesty, and that guilt (from Hat being a wild success) and changed methodology (his fame meant a lot of spontaneous contacts so he didn't need to make anything up to have something to write about nor did he have any concern about getting published or sales) mean that the later books were probably far more trustworthy (but also m... (read more)

I did say roughly that this would happen, the thing I regret was somehow threading the needle on communicating:

"guys, when I say it's gonna get dark, I'm, like, more serious than usual. But I am also more serious than usual about it being light at the end, when you are evaluating the darkness which is darker than you expect, I will be trying pretty hard to counterbalance that." (literally those words feels too awkward, but, that's the vibe I wanted)

Probably the biggest mistake I made was not emphasizing at the beginning of the darkness "Guys, this may be intense, and I'm not sure how this will go for everyone. But, I have worked hard to ensure Solstice leaves you better than it found you. I wouldn't have done tonight the way I did if I didn't think I had."

NYC's 2024 winter solstice was my first solstice so I roughly knew to expect things to get sad/dark and then to get more positive after, but I don't recall if there was a slide or description of that in the beginning. my memory feels pretty fuzzy here, so it's entirely possible this was covered and I just forgot! 

We implemented Anthropic's JumpReLU training method in SAELens - give it a try!

This link is broken; may have to do with your migration to decoderesearch.

Edit: I'm pretty sure the correct link is now here

I agree it is technically an evaluation of our cost-effectiveness! Though not much more than generic insults are, which I wouldn't feel much of a need to link. I think it was reasonable for you to link it here, and also don't think it makes sense to include it in the post. 

Running Less Wrong doesn't (have to) cost 1.4M per year. It's a website. Websites of this scale can be run at 1/10th the cost, or even 1/100th the cost if you're frugal. Having extremely onerous community management practices is a choice you're making.

This is just really not true. L... (read more)

As far as I'm aware of, this is one of the very few pieces of writing that sketches out what safety reassurances could be made for a model capable of doing significant harms. I wish there were more posts like this one. 

Strongly agree. Sell-price tax with forced sales sounds like something a cryptocurrency would implement. It might work there, since if a malicious bidder tried to buy your TOKEN at above-market price, you could automatically buy a new one within the same block, at actual-market price. This could also work for fungible but rapidly-transferrable assets like cloud GPU time.

If taxing physical goods (like infrastructure or even land) which is where a lot of value in the world lies, this does just open up companies for extortion. E.g. what if I demand to buy one... (read more)

This was my favourite solstice to date. Thank you.

What I linked was a stance, an evaluation of your cost-effectiveness. You were explicitly asking for exactly that kind of thing.

But on 2nd thought, I retract the link I posted. There's also quite a bit of sneering in there that distracts from the bits I consider important, which are these:

1. Running Less Wrong doesn't (have to) cost 1.4M per year. It's a website. Websites of this scale can be run at 1/10th the cost, or even 1/100th the cost if you're frugal. Having extremely onerous community management practices is a choice you're making.

2. If someone bel

The post "Butterfly Ideas" seems relevant.

I wouldn't make that argument. I just don't see the point of keeping it real.

It just seems like going virtual opens up a lot of possbilities with no downside. If you want consistency and real work, put it in your sim. Share it with other real people if you want to compromise on what worlds you'll inhabit and what challenges you'll face.

If you want real people who are really suffering so there are real stakes to play for, well, that's an orthogonal issue. I'd rather see nonconsensual suffering eliminated.

So: Why prefer the Real? What's it got that the Virtual doesn't?

I'm not sure what you mean. You can list something at market price rather than how you actually value it, but that opens you up to significant risk from market volatility and malicious bids (from competitors looking to cripple your business, rich people you've offended, etc). If sales cannot actually be forced this seems more workable, but still exploitable by malicious actors.

Trying to align LLMs just doesn't seem optional to me. It's what's happening whether or not we like it.

The costs being part of the outcome is an interesting point. But the example is maybe weaker than it first seems. If both sides knew perfectly in advance that the cost of taking Eastern Examplestan by force is 300,000 fighting-aged males, then the aggressor could probably find something that they valued less than 300,000 of their own lives, but that the defender valued more highly than their deaths. Eg. The aggressor might think paying $100 -million was better than all that death, and the defender might rather have $100 million than to inflict those casual... (read more)

Makes sense! Options like giving what we can regranting seem to work for a bunch of people but maybe they aren't willing to do it for lightcone?

GWWC wasn't accepting new applications for organizations to sponsor this way last time I checked a few months ago. I asked them to let me know when they do, and haven't heard back, but I should ping them again and double check!

Also, to give a bit more context on my thinking here, I currently think that it's fine for us to accept funding from Deepmind safety employees without counting towards this bucket, largely because my sense is the social coordination across the pond here is much less intense, and generally the Deepmind safety team has struck me as the most independent from the harsh financial incentives here to date

Seems true to me, though alas we don't seem likely to be getting an Anthropic level windfall any time soon :'(

Makes sense! Options like giving what we can regranting seem to work for a bunch of people but maybe they aren't willing to do it for lightcone?

But yeah, any donor giving $50K+ can afford to set up their own donor advised fund and get round this kind of issue, so you probably aren't losing out on THAT much by having this be hard.

Yep, agree that it's better to be funded by a collection of 20+ disagreeable frontier lab employees than 100% Open Phil, but I do also pretty intensely regret relying so much on Open Phil funding, so this is compatible with my state of belief. 

I think social group beliefs are generally very correlated with each other, and think it will likely be the case that if we rely very heavily on frontier lab employee funding that there will be a lot of implicit pressure that will be backed by a threat of successful coordination to not fund us if we disagree.&nb... (read more)

What are the specific concerns re having too much funding come from frontier lab employees? I predict that it's much more ok to be mostly funded by a collection of 20+ somewhat disagreeable frontier lab employees who have varied takes, than to have all come from OpenPhil. It seems much less likely that the employees will coordinate to remove funding at once or to pressure you in the same way (especially if they involve people from different labs)

Ah, this is a great point. I will update the text of the article with some guidance to this effect. 

Agree it would be good to set up some better way for small to medium UK based donors to donate tax deductibly, but haven't found a good way of doing so that wouldn't cost me 40+ hours.

I think that number is way too low for anyone who OpenAI actually really cares about hiring. Though this kind of thing is very very heavy tailed

If you are in the UK, donate through NPT Transatlantic.

This recommendation will not work for most donors. The fees are a flat fee of £2,000 on any donation less than £102,000. NPT are mostly a donor advised fund service. While they do allow for single gifts, you need to be giving in the tens of thousands for it to make any sense.

The Anglo-American Charity is a better option for tax deductible donations to US charities, their fee is 4% on donations below £15K minimum fee £250 (I have not personally used them, but friends of mine have)

Given the size of th... (read more)

That makes sense! (That comment is replying to what seems like a different claim that seems more obviously wrong than yours though.)

If you're talking about surreals or hyperreals, the issue is basically that there's not one canonical model of infinitesimals, you can create them in many different ways. I'll hopefully end up writing more about the surreals and hyperreals at some point, but they don't solve as many issues as you'd hope unfortunately, and actually introduce some other problems.

As a motivating idea here, note that you need the Boolean Prime Ideal Theorem (which requires a weak form of choice to prove) to show that the hyperreals even exist in the first place, if you're star... (read more)

I was trying to note that the answers are bounded above too, and in this particular case we can infer that at least a quarter of Americans have insane takes here. (Though the math I did was totally wrong.)

The effect isn't instantaneous, right? Is a moment in the HVAC system enough to kill them?

The Australian financial year starts and ends in the middle of the year, so it makes no difference to me if we do it in 2026. Let's make it happen :)

Sorry, you're totally right.

There are some similarities between autism and ADHD that are probably worth exploring, because people can do similar things for different reasons, and then the same solutions may not apply.

For example, if I understand it correctly, ADHD makes it difficult to focus on important things unless they are also interesting. So you can help by e.g. introducing gamification. Autism makes it difficult to do anything when you are overwhelmed by stimuli. Gamification will probably only make it worse, because it is one extra stimulus.

Where a person with ADHD may benefit from a pill, an autist may benefit from a quiet work environment, etc.

I think a lot of people got baited hard by paech et al's "the entire state is obliterated each token" claims, even though this was obviously untrue even at a glance

A related true claim is that LLMs are fundamentally incapable of introspection past a certain level of complexity (introspection of layer n must occur in a later layer, and no amount of reasoning tokens can extend that), while humans can plausibly extend layers of introspection farther since we don't have to tokenize our chain of thought.

But this is also less of a contraint than you might expect when frontier models can have more than a hundred layers (I am an LLM introspection believer now).

what’s the principle here? if an agent would have the same observations in world W and W’ then their preferences must be indifferent between W and W’ ? this seems clearly false.

No? The entire post is about how the average estimate is computed using the arithmetic mean, so you can be skewed by a small % of respondents giving very high estimates. Maybe I'm missing something though.

Now, where did the weirdness come from here. Well, to me it seems clear that really it came from the fact that the reals can be built out of a bunch of shifted rational numbers, right?

I think the weirdness comes from trying to assign a real number measure, instead of allowing infinitesimals. I've never understood why infinite sets are readily accepted, but infinitesimal/infinite measures are not.

EDIT: To explain my reasoning more, suppose you were Pythagoras and your student came to you and drew a geometric diagram with lengths not in a ratio of whole n... (read more)

I like this post! The evidence was presented well in a non-judgemental manner (neither bashing respondents or pollsters), and it changed my mind by emphasising my previously understood-but-not-intuitive knowledge how outliers can shape means.

When I read the title, I thought you were going to talk about how LLMs sometimes claim bodily sensations such as muscle memory. I think these are probably confabulated. Or at least, the LLM state corresponding to those words is nothing like the human state corresponding to those words.

Expressions of emotions such as joy? I guess these are functional equivalents of human states. A lack of enthusiasm (opposite of joy)  an be reflected in the output tokens.

No. The average estimate is 26%, which implies at least 26% of the polled population give an estimate of 26% or higher, i.e. a very large gravel of respondents are either very confused or intentionally giving inflated answers.

Yeah, I definitely don't think the underlying states are exactly identical to the human ones! Just that some of their functions are similar at a rough level of description.

(Though I'd think that many humans also have internal states that seem similar externally but are very different internally, e.g. the way that people with and without mental imagery or inner dialogue initially struggled to believe in the existence of each other.)

In most of these examples, LLMs have a state that is functionally like a human state, e.g. deciding that they’re going to refuse to answer, or “wait…” backtracking in chain of thought. I say Functionally, because these states have externally visible effects on the subsequent output (e.g. it doesn’t answer the question). It seems that LLMs have learned the words that humans use for functionally similar states (e.g, “Wait”).

The underlying states might not be exactly human identical. “Wait” backtracking might have function differences from human reasoning that are visible in the tokens generated.

If you place the lamps so they're only above people's heads you can use 254nm bulbs, which are much cheaper (they're essentially standard fluorescent lights with UV-transparent glass and no phosphor). People have done this for a long time in places like TB wards, but you do need to be pretty careful about placement to ensure your 254nm UV really is only shining in the space above.

Of course the ideal method is to have the UVC light internal to your air conditioner/heater unit, which is already circulating the air, so you can blast everything passing through that with enough UVC to annihilate any and all pathogens in the air, but that requires retro-fitting to AC units and stuff.

To get equivalent protection this way you'd need to cycle your air through your HVAC much faster than you likely currently do. Which would be noisy!

Yes sorry to be clear I’m not talking about whether it is true, I am talking about whether they would use it or not in proving ‘standard’ results.

Yeah, I think some rationalists, e.g. Eliezer, use it a lot more than the general population, and differently from the popular figurative sense. As in "raising the sanity waterline."

I would note that by the Markov inequality, at least 25% of Americans must think that foreign aid is more than 25% of the budget in order to get the average response we see here.

Isn't it more like at least ~1.3% of Americans must think that foreign aid is more than 25% of the budget? The extreme case here is where p% think it's 100% and (1-p)% think it's exactly 25%, which comes out to p=~1.3%. 25% only seems right if (1-p)% guess 0% and p% guess 100%.

I tried to search for surveys of mathematicians on the axiom of choice, but couldn't find any. I did find one survey of philosophers, but that's a very different population, asked whether they believed AC/The Continuum Hypothesis has an answer rather than what the answer is:  https://thephilosophyforum.com/discussion/13670/the-2020-philpapers-survey

My subjective impression is that my Mathematician friends would mostly say that asking whether AC is true or not is not really an interesting question, while asking what statements depend on it is.

this seems to assume that consciousness is epiphenomenal.

To my understanding, epiphenomenalism is the belief that subjective consciousness is dependent on the state of the physical world, but not the other way around. I absolutely do not think I assumed this - I stated that it is either true ("If consciousness does not have a determinative effect on behavior,") or it is not ("If consciousness has a determinative effect on behavior,"). The basis of my claim is a proof by case which aims to address both possibilities.

That... I think is below my bar for having any actual logical content instead of just snark, but if a few people want to second that they would have somehow found this valuable to read, I can add it to the post.

An underrated answer is that humans are very, very dependent on other people to survive, and we have easily the longest childhood where we are vulnerable of any mammal, and even once we do become an adult, we are still really, really bad at surviving on our own compared to other animals, and since we are K-selected, every dead child mattters a lot in evolution, so it's very, very difficult for sociopathy to be selected for.

how to set up some kind of good quality AI Safety Forum that Coefficient is willing to fund. 

There are millions of non-OP dollars in the ecosystem. Even without capability money, trying to start a new forum because of some political preferences by Dustin seems both dumb on a tactical level, and dumb on an incentive level (the last thing we want is for infrastructure projects to be downstream of random US political drama).

how likely do you think it is that after the IPO a few rich Anthropic employees will just cover most of Lightcone's funding need?

I d... (read more)

More speculative thoughts:

• Perhaps we end up in a regime where this is possible, but expensive enough that we can only afford a bit of it.
• If that's the case, the more sample-efficient the RL is, the better off we are — since as the total number of RL rewards we need decreases, we can afford to invest more in the quality of each one.

Even more speculative thoughts:

• As an extreme example, you could imagine an AI that's as sample-efficient as a human child. Suppose an AI company had a virtual copy of von Neumann's brain age 5, which they could run at 10x sp

Changing selection pressures to align with intended behaviors: This might involve making training objectives more robust, iterating against held-out evaluation signals, or trying to overwrite the AI's motivations at the end of training with high-quality, aligned training data.

Is another broad direction for this increasing the intelligence of the reward models? A fun hypothetical I've heard is to imagine replacing the reward model used during training with Redwood Research (the organization).

So we imagine that during training, whenever an RL reward must be ... (read more)

Good thing you're only incentivized to sell at market price then, not how much value you'd assign personally!

An AI content X/Twitter account with nearly 100k followers blocked me, and I got a couple of disapproving replies for pointing out that the account was AI-generated. I quote-tweeted the account mostly to share a useful Chrome Extension that I've been using the detect AI content, but I was surprised that there was a negative reaction in the form of a few replies pointing out the account was AI-generated. I am neither pro- nor anti-AI accounts, but being aware of the nature of the content seems to be useful.

Would be curious to hear others' thoughts on the ph... (read more)

Oh, yeah sure. I mean I think as a matter of pragmatics it mostly is an exercise in foundations these days. But I agree that splitting up concepts of finitude -- for example -- is a super interesting investigation. Just, like, the majority of algebraic geometers, functional analysts, algebraic topologists, analytic number theorists, algebraic number theorists, galois theorists, representation theorists, differential geometers, etc etc etc, would not be all that interested in such an investigation these days.

Agreed on all points. Including that you writing a more detailed version more for the prosaic crowd isn't probably the best next step. That's what I was trying to do in LAMRAG, and it was no more successful than this. That's despite me starting much closer to the standard prosaic alignment/LLM-based model of AGI internals.

I think one place this argument may break down for people is the metaphor of building for the ocean as a difficult project. Maybe the lake is a lot like the ocean, and ocean storms just aren't that bad, so you can just build it to double ... (read more)

Sorry, I think explaining without using type theory what you are trying to say may help me understand better?

EDIT: like, in particular, insofar as its relevant to the axiom of choice.

I think among working mathematicians it is that much of a minority. E.g. it is much less controversial than something like many-worlds among physicists, or something like heritability of intelligence among geneticists. It is broadly incredibly accepted.

EDIT: But I do agree it's a respected alternative view, and I do think it should stay that way because it is interesting to investigate. I just think people get the wrong idea about its degree of controversy among working mathematicians.

Committing to writing 1 post a week has been a really good practice for me. Although I decided to take December off for a variety of reasons, the need to keep publishing has forced me to become a more efficient writer, to write about topics I might otherwise have neglected, and of course to get more feedback on my writing than I otherwise would if I spent more time effortposting and less time actually posting. I expect to keep it up in 2026.

None of the evidence or framing here is particularly worse than what I would expect from a biased against LW search & retelling of the history, and that is what you're getting if you read Bouchard's writing, so this seems like effectively zero evidence.

It is also notable that they have nothing good to say about LessWrong's response. I think at minimum you have to credit the trading gains many LessWrongers made at the time, and MicroCovid.

That is not to say they are wrong about how early LessWrong was, just that it provides extremely little evidence. Be... (read more)

What's your take on why Approval Reward was selected for in the first place VS sociopathy?

I find myself wondering if non-behavioral reward functions are more powerful in general than behavioral ones due to less tendency towards wireheading, etc.(consider the laziness & impulsivity of sociopaths) Especially ones such as Approval Reward which can be "customized" depending on the details of the environment and what sort of agent it would be most useful to become.

If you can't tell the difference, how could you care which is which?

I'm talking about blocking your memories of living in a simulated world.

Consciousness is very unlikely to be a binary property. Most things aren't. But there appears to be a very strong tendency for even rationalists to make this assumption in how they frame and discuss the issue.

The same is probably true of moral worth.

Taken this way, LLMs (and everything else) are partly conscious and partly deserves moral consideration. What you consider consciousness and what you consider morally worthy are to some degrees matters of opinion, but they very much depend on facts about the minds involved, so there are routes forward.

IMO curre... (read more)

And I guess having the UVC in ducts doesn't disinfect surfaces in rooms, so you'd need a separate solution for that.

Yes, that visual diagram is very helpful. Thank you! I think I mostly get it now.

I think your title "The Axiom of Choice is Not Controversial" is literally false, because "controversial" is a social property referring to how much disageement there is, and not whether something is true. Rejecting the AoC is a minorinty view but it is not that minority. It's also a respected alternative view, and texts often go out of their way to mention when they are acceping the AoC. It may if anything be the most controversial thing in math.

Perhaps it is natural and good populations bind together when threatened existentially, and fracture when wealthy and peaceful.

Perhaps it is good among humans, and I do think it is natural for us, but FWIW I think it's a mistake to consider this a problem with liberalism or liberal democracy specifically or especially, even though that's a relevant context today. The phrase "The empire divided longs to unite, united longs to divide" comes from China, after all.

Hm. I think that all makes sense.

Now I'm wondering whether specificity can be measured in a sort of absolute sense rather than in a relative sense.

You mention that sunny day is more specific than day because it adds weather. Or as others have mentioned, because the set of points it includes is a subset of the set of points that day includes.

But what about the concept of "pizza that is warm, has an exterior that is thin and crispy, an interior that is warm, chewy, and fresh, a thin layer of tomato sauce that is mildly sweet and acidic, and small dollups of ... (read more)

What about now? It is almost 2026 and the Singularity is nearer than before and it would make sense to me that maybe its not on the critical path for anything urgent, but... <3

I'm not making a one-time donation to Lighthaven, because I already have a recurring donation set up to Lighthaven. I encourage doing this as a general best practice for those donating ordinary USD or similarly boring funds, especially if your donation approach isn't set up for a lump-sum approach.  

While obviously less valuable in net present value terms than donating the full value upfront, having a predictable recurring fundraising flow from donors is quite useful for nonprofits.

This has a lot of overlap with my recent post LLM AGI may reason about its goals and discover misalignments by default, and the followup post I'm working on now that further explores whether we should train LLMs on reasoning about their goals. Prompting them to reason extensively about goals during training has the effect of revealing potential future misalignments to them, as you discuss. 

I'm curious what you think about that framing.

The signaling model of war is valuable but can be taken too far: if leadership focuses too much on the signals they're sending, that can get in the way of actually fighting the war; Vietnam is often pointed to as an example of this. From Friedman's The Fifty Year War:

Soon Johnson began a program of continuous air attacks against North Vietnam, Rolling Thunder, which would last through March 1968. In line with current nuclear escalation ideas, Johnson and McNamara treated air attacks as messages to the North Vietnamese. They personally selected the targets

This very roughly implies that the median of "50% time horizon as predicted by METR staff" by EOY 2026 is a bit higher than 20 hours.

If you follow maths, one can be reasonably confident that the models can now sometimes solve math problems that are "not too hard in retrospect". I don't know how substantial this particular problem was supposed to be, but it feels like it tracks?

I feel like reporting the median is much simpler than these other proposals, and is probably what should be adopted.

I would note that by the Markov inequality, at least 25% of Americans must think that foreign aid is more than 25% of the budget in order to get the average response we see here. So I think it's reasonable to use the reported mean to conclude that at least a sizable minority of Americans are very confused here.

I very roughly polled METR staff (using Fatebook) what the 50% time horizon will be by EOY 2026, conditional on METR reporting something analogous to today's time horizon metric.

I got the following results: 29% average probability that it will surpass 32 hours. 68% average probability that it will surpass 16 hours. 

The first question got 10 respondents and the second question got 12. Around half of the respondents were technical researchers. I expect the sample to be close to representative, but maybe a bit more short-timelines than the rest of METR staff.

The average probability that the question doesn't resolve AMBIGUOUS is somewhere around 60%.

Indeed, the traffic for AI 2027 in 2025 has been similar to the traffic for all of LessWrong in 2025, with about 5M unique users and 10M page-views, whereas LessWrong had 4.2M unique users and 22M page-views. 

While my inner consultant loves this approach to impact sizing, I think another approach is to ask how much various political efforts would pay to get the Vice President (which on rough historical odds has ~50% chance of being the next President, in addition to his current powers) to read and publicly say that he'd read a document and had a positive opinion of it.

It is, uh, more like the think tank number than the paid ads number.

If safety is a concern for such sources, is it worth considering placing the lights so they mostly shine on the space above people's heads?

More reasons:

2.b. The problem is not lack of legible evidence per se, but the fact that the other members of the group are too stupid to understand anything; from their perspective even quite obvious evidence is illegible.

7. If you attack them and fail, it will strengthen their position; and either the chance of failure or the bonus they would get is high enough to make the expected value of your attack negative.

For example, they may have prepared a narrative like "there is a conspiracy against our group that will soon try to divide us by bringing up unfounded accusations against people like me", so if your fail to convince the others, you will provide evidence for the narrative.

I think you misinterpreted me - my claim is that working without choice often reveals genuine hidden mathematical structures that AC collapses into one. This isn't just an exercise in foundations, in the same way that relaxing the parallel postulate to study the resulting inequivalent geometries (which were equivalent, or rather, not allowed under the postulate) isn't just an "exercise in foundations."

Insofar as [the activity of capturing natural concepts of reality into formal structures, and investigating their properties] is a core part of mathematics, the choice of working choice-free is just business as usual.

Do you have a specific plan, or is this just a call to signal virtue by doing costly unhelpful actions?

This post and (imo more importantly) the discussion it spurred has been pretty helpful for how I think about scheming. I'm happy that it was written!

Yeah that seems to be the most serious one, and the only one I could see that I had a real issue with.

I’d be happy to swap my 1.5k to Lightcone for your 1.5k to Givewell All Grants.

Practical note: For my own tax reasons, I’d be looking to do this at the start of 2026 (but within Lightcone’s fundraising window of course). This doesn’t feel too hard to coordinate around even if you need to give in 2025 for your tax reasons, but might be relevant for comparison of offers.

foo :: $(Int\to \perp )\to \perp$
foo f = f 4

Look, there's an integer! It's right there, "4". Apparently $Int$ is inhabited.

bar :: $\left(\right(a\to \perp )\to \perp )\to a$
bar fo = ???

There's nothing in particular to be done with fo... if we had something of type $a\to \perp$ to give fo, we would be open for business, but we don't know enough about $a$ to make this any easier than coming up with a value of type $\perp$, which is a non-starter.