forecasting is so intractible-in-practice that metaculus doesn't even use it for internal decisions, despite attempting to do so multiple times (source: i was there)
XBOW's own "Top 1" announcement is dated June 24, 2025--about 8 months before this LessWrong post (Feb 19, 2026), not "almost one year."
this is borderline pedantry, and also the timing of the announcement is not significant to the thesis of the post. if your goal is to tell us "here's what the extension is like, also be aware that some of the corrections are wrong/unhelpful" then fine, but if it's a sales pitch you should choose a better example.
this is a neat article and all, but what are the actual sources for any of this? what is the author's relation to this story? they slide between statistics and checkable facts smoothly into anecdata of "this is the general sort of thing that happens in places like these"
But there’s an additional problem here. People overestimate foreign aid, and these polling methods exaggerate how much they overestimate. In the KFF data, the majority of respondents said it was below 20%, and the largest decile by far was 0-10%. KFF said that average was 26%, but their own data show that this isn’t what most people actually believe; it's an artifact of using the wrong statistic.
the KFF data shows that the median respondent thinks foreign aid is somewhere in the 11-20% of the budget, so reporting the average is roughly 2.4 to 1.3 tim...
Jim, Garfield's owner,
Garfield's owner's name is Jon Arbuckle. the cartoonist/author of Garfield is Jim Davis
i notice the OP didn't actually mention examples of legible or illegible alignment problems. saying "leaders would be unlikely to deploy an unaligned AGI if they saw it had legible problem X" sounds a lot like saying "we would never let AGI onto the open internet, we can just keep it in a box", in the era before we deployed sydney soon as it caught the twinkle of a CEO's eye.
your entire analysis is broken in that you assume that an elo rating is something objective like an atomic weight or the speed of light. in reality, an elo rating is an estimation of playing strength among a particular pool of players.
the problem that elo was trying to solve was, if you have players A and B, who have both played among players C through Q, but A and B have never played each other, can you concretely say whether A is stronger than B? the genius of the system is that you can, and in fact, the comparison of 2 scores gives you a probability o...
i think you're mis-applying the moral of this comic. the intended reading IMO is "a person believes misinformation, and perhaps they even go around spreading the misinformation to others. when they've been credibly corrected, instead of scrutinizing their whole ideology, they go 'yeah but something like it is probably true enough'." OP doesn't point to any names or say "this is definitely happening", they're speculating about a scenario which may have already happened or may happen soon, and what we should do about it.
Though, notably, Metaculus lists Jan 2027 as a "community prediction" of "weakly general AI". Sure, someone could argue that weakly general AI doesn't imply human-level AGI soon after
it does imply that, but i'm somewhat loathe to mention this at all, because i think the predictive quality you get from one question to another varies astronomically, and this is not something the casual reader will be able to glean
...The True Believers hypothesis rings false because that would be a frankly ridiculous belief to hold. Sometimes people profess ridiculous things, but very few of them put their money where their mouth is on prediction markets. [1]
- I’ve seen some pretty mispriced markets. At one point in 2019, PredictIt had Andrew Yang at 16% to win the Democratic presidential primary. And in 2020, Donald Trump was about 16% to become president even after he had lost the election. But the sorts of people who bet on prediction markets are not the sorts of fundamentalist Ch
this is a fair response, and to be honest i was skimming your post a bit. i do think my point somewhat holds, that there is no "intelligence skill tree" where you must unlock the level 1 skills before you progress to level 2.
i think a more fair response to your post is:
Surely it would be exceptionally good at those kinds of writing, too, right?
surely an LLM capable of writing A+ freshman college papers would correctly add two 2-digit numbers? surely an AI capable of beating grandmasters in chess would be able to tutor a 1000 elo player to a 1500 elo or beyond? surely an AI capable of answering questions at a university level in diverse subjects such as math, coding, science, law, would be able to recursively improve itself and cause an intelligence explosion? surely such an AI would at least be ab...
In some sense, the Agent Foundations program at MIRI sees the problem as: human values are currently an informal object. We can only get meaningful guarantees for formal systems. So, we need to work on formalizing concepts like human values. Only then will we be able to get formal safety guarantees.
unless i'm misunderstanding you or MIRI, that's not their primary concern at all:
...Another way of putting this view is that nearly all of the effort should be going into solving the technical problem, "How would you get an AI system to do some very modest con
this was posted after your comment, but i think this is close enough:
And the idea that intelligent systems will inevitably want to take over, dominate humans, or just destroy humanity through negligence is preposterous.
They would have to be specifically designed to do so.
Whereas we will obviously design them to not do so.
you're ascribing too much consequentialism to people who are generally not consequentialists. the mindset is more like: