Fun Theory is the study of questions such as "How much fun is there in the universe?", 
"Will we ever run out of fun?", "Are we having fun yet?" and "Could we be having 
more fun?". It's relevant to designing utopias and AIs, among other things.

If you want to better understand counting arguments for deceptive alignment, my comment here might be a good place to start.
It seems slightly bad that you can vote on comments under the Quick Takes and Popular Comments sections without needing to click on the comment and without leaving the frontpage. This is different than posts where in order to vote, you have to actually click on the post and leave the main page (which perhaps requires more activation energy). I'm unsure, but I think this could result in a higher frequency of cases where Quick Takes and comments go viral in problematic ways relative to posts. (Based on my vague sense from the EA forum.) This doesn't seem very important, but I thought it would be worth quickly noting.
Marginally against legibilizing my own reasoning:      When taking important decisions, I spend too much time writing down the many arguments, and legibilizing the whole process for myself. This is due to completionist tendencies. Unfortunately, a more legible process doesn’t overwhelmingly imply a better decision! Scrutinizing your main arguments is necessary, although this looks more like intuitively assessing their robustness in concept-space than making straightforward calculations, given how many implicit assumptions they all have. I can fill in many boxes, and count and weigh considerations in-depth, but that’s not a strong signal, nor what almost ever ends up swaying me towards a decision! Rather than folding, re-folding and re-playing all of these ideas inside myself, it’s way more effective time-wise to engage my System 1 more: intuitively assess the strength of different considerations, try to brainstorm new ways in which the hidden assumptions fail, try to spot the ways in which the information I’ve received is partial… And of course, share all of this with other minds, who are much more likely to update me than my own mind. All of this looks more like rapidly racing through intuitions than filling Excel sheets, or having overly detailed scoring systems. For example, do I really think I can BOTEC the expected counterfactual value (IN FREAKING UTILONS) of a new job position? Of course a bad BOTEC is better than none, but the extent to which that is not how our reasoning works, and the work is not really done by the BOTEC at all, is astounding. Maybe at that point you should stop calling it a BOTEC.
I often wish I had a better way to concisely communicate "X is a hypothesis I am tracking in my hypothesis space". I don't simply mean that X is logically possible, and I don't mean I assign even 1-10% probability to X, I just mean that as a bounded agent I can only track a handful of hypotheses and I am choosing to actively track this one. * This comes up when a substantially different hypothesis is worth tracking but I've seen no evidence for it. There's a common sentence like "The plumber says it's fixed, though he might be wrong" where I don't want to communicate that I've got much reason to believe he might be wrong, and I'm not giving it even 10% or 20%, but I still think it's worth tracking, because strong evidence is common and the importance is high. * This comes up in adversarial situations when it's possible that there's an adversarial process selecting on my observations. In such situations I want to say "I think it's worth tracking the hypothesis that the politician wants me to believe that this policy worked in order to pad their reputation, and I will put some effort into checking for evidence of that, but to be clear I haven't seen any positive evidence for that hypothesis in this case, and will not be acting in accordance with that hypothesis unless I do." * This comes up when I'm talking to someone about a hypothesis that they think is likely and I haven't thought about before, but am engaging with during the conversation. "I'm tracking your hypothesis would predict something different in situation A, though I haven't seen any clear evidence for privileging your hypothesis yet and we aren't able to check what's actually happening in situation A." * A phrase people around me commonly use is "The plumber says it's fixed, though it's plausible he's mistaken". I don't like it. It feels too ambiguous with "It's logically possible" and "I think it's reasonably likely, like 10-20%" and neither of which is what I mean. This isn't a claim about its probability, it's just a claim about it being "worth tracking". Some options: * I could say "I am privileging this hypothesis" but that still seems to be a claim about probability, when often it's more a claim about importance-if-true, and I don't actually have any particular evidence for it. * I often say that a hypothesis is "on the table" as way to say it's in play without saying that it's probable. I like this more but I don't feel satisfied yet. * TsviBT suggested "it's a live hypothesis for me", and I also like that, but still don't feel satisfied. How these read in the plumber situation: * "The plumber says it's fixed, though I'm still going to be on the lookout for evidence that he's wrong." * "The plumber says it's fixed, though it's plausible he's wrong." * "The plumber says it's fixed, and I believe him (though it's worth tracking the hypothesis that's he's mistaken)." * "The plumber says it's fixed, though it's a live hypothesis for me that he's mistaken." * "The plumber says it's fixed, though I am going to continue to privilege the hypothesis that he's mistaken." * "The plumber says it's fixed, though it's on the table that he's wrong about that." Interested to hear any other ways people communicate this sort of thing! Added: I am reacting with a thumbs-up to all the suggestions I like in the replies below.
Remember, they are not "hallucinations", they are confabulations produced by dream machines i.e. the LLMs!

Popular Comments

Recent Discussion

The comments here are a storage of not-posts and not-ideas that I would rather write down than not.

Before I read the aphoristic three-word reply to you from Richard Kennaway (admittedly a likely even clearer-cut way to indicate the following sentiment), I was thinking that to downplay any unintended implications about the magnitude of your probabilities that you could maybe say something about your tracking being for mundane-vigilance or intermittent-map-maintenance or routine-reality-syncing / -surveying / -sampling reasons.

For any audience you anticipate familiarity with this essay though, another idea might be to use a version of something like:

"The ... (read more)

Bridge, like most card games, is a game of incomplete information. It is a game of many facets, most of which will have to remain unstated here. However the calculation of probabilities, and the constant revision of probabilities during play, is a key feature. Much the same can be said of poker and of certain other card games. But bridge is the game I know and I am still learning from, 45 years after I first got infected with the bridge bug

We have Alpha Go and Alpha Zero but the wizards at DeepMind have not yet seen fit to release an Alpha Bridge. Poker is played to high expert standard by specialist AI’s such as Pluribus but bridge  playing computers up till today remain well short of...

I'm pretty dissapointed by the state of AI in bridge. IMHO the key milestones for AI would be:
1) Able to read and understand a standard convention card and play with/against that convention.
2) Decide the best existing convention.
3) Invent new, superior conventions. This is where we should be really scared. claims that 1 bit LLMs are possible.

If this scales, I'd imagine there is a ton of speedup to unlock since our hardware has been optimized for 1 bit operations for decades. What does this imply for companies like nvidia and the future of LLM inference/training? 

 Do we get another leap in LLM capabilities? Do CPUs become more useful? And can this somehow be applied to training?

Or is this paper not even worth considering for some obvious reason I can't tell. 

I don't think it can be patched for training (95% confident). I think training (not inference) is where most the money goes to and comes from, so hardware market will not be affected (90%).

Even in the small inference market, chip companies already have 4-8 bit inference accelerators in the oven (99%); they will not estimate the benefits of 1.58 bit to justify the risk of such specialized hardware, so nobody will build more than 100 1-bit or 1.58-bit inference chips (80%).

Old fashioned CPUs have at most 32 threads so will still be slow as heck to run NNs (90%).

I think your question is quite important.


I have compiled a list of possible future scenarios. I hope this list is useful in two ways:

  • As a way to make your own thinking about the future more explicit; how much probability mass do you put on each possible future? 
  • As a menu of options to choose from; which of these futures do we want to make more likely?

This list is just a brainstorm, and I encourage readers to write any missing but probable futures in the comments. I will add any scenarios that do not substantially overlap with existing items and which I subjectively estimate as having at least a 0.01% probability of happening to the list (with attribution).

I have divided the possible futures into the following categories:

  • Futures without AGI, because we prevent building it
  • Futures without

Thanks for the suggestion! @BeyondTheBorg suggested something similar with his Transcendent AI. After some thought, I've added the following:

Transcendent AI: AGI uncovers and engages with previously unknown physics, using a different physical reality beyond human comprehension. Its objectives use resources and dimensions that do not compete with human needs, allowing it to operate in a realm unfathomable to us. Humanity remains largely unaffected, as AGI progresses into the depths of these new dimensions, detached from human concerns.


The 5th Annual LessWrong Review has come to a close!

Review Facts

There were 5330 posts published in 2022. 

Here's how many posts passed through the different review phases.

PhaseNo. of postsEligibility
Nominations Phase579Any 2022 post could be given preliminary votes
Review Phase363 Posts with 2+ votes could be reviewed
Voting Phase168Posts with 1+ reviews could be voted on

Here how many votes and voters there were by karma bracket.

 Karma BucketNo. of VotersNo. of Votes Cast 

To give some context on this annual tradition, here are the absolute numbers compared to last year and to the first year of the LessWrong Review.

 Total LW Posts170345065330 

Review Prizes

There were lots of great reviews this year! Here's a link to all of them

Of 227 reviews we're giving 31 of them prizes. 

This follows up on Habryka who gave out about half of...

The new designs are cool, I'd just be worried about venturing too far into insight porn. You don't want people reading the posts just because they like how they look (although reading them superficially is probably better than not reading them at all). Clicking on the posts and seeing a giant image that bleeds color into the otherwise sober text format is distracting. 

I guess if I don't like it there's always GreaterWrong.

Cross-post of my blog article on the topic.  

I probably know less about science than most people who think the earth is flat do.

Okay, that’s not quite true. I have knowledge of lots of claims of science—that the big bang happened, that evolution is true, that the earth is round, etc—that people who think the earth is flat don’t have. But in terms of my knowledge of the science surrounding how one shows the roundness of the earth, I, like most people who think the earth is round, probably know less about it than most people who think the earth is flat.

People are often baffled by the widespread success of conspiracy theories. Among well-educated liberals, for instance, there’s a common attitude that conspiracy theorists are just deeply...

I've been planning to write a post around the same lines. Well done.

Due to a historically terrible name, people assume that conspiracy theories are about the existence of conspiracies. That everything that supposes that there may be a conspiracy - is a conspiracy theory. This makes such reference class extremely unhelpful. There have always been actual conspiracies. People lie and plot and conceal information. We can't put all the actual examples of conspiracies in the same category as Flat Earth and treat the whole category as a priory implausiable.

What c... (read more)

See also "Nevertheless, the model of misinformation as a societal disease often gets things backwards. In many cases, false or misleading information is better viewed as a symptom of societal pathologies such as institutional distrust, political sectarianism, and anti-establishment worldviews. When that is true, censorship and other interventions designed to debunk or prebunk misinformation are unlikely to be very effective and might even exacerbate the problems they aim to address. To begin with, the central intuition driving the modern misinformation panic is that people—specifically other people—are gullible and hence easily infected by bad ideas. This intuition is wrong. A large body of scientific research demonstrates that people possess sophisticated cognitive mechanisms of epistemic vigilance with which they evaluate information. If anything, these mechanisms make people pig-headed, not credulous, predisposing them to reject information at odds with their pre-existing beliefs. Undervaluing other people’s opinions, they cling to their own perspective on the world and often dismiss the claims advanced by others. Persuasion is therefore extremely difficult and even intense propaganda campaigns and advertising efforts routinely have minimal effects."
Conspiracy theories are a bad reference class due to the lumping together of real actions by nation-states with crackpot schizophrenic fantasies. This was intentional and you shouldn't buy into it.
As Mark Twain supposedly once said: it's not what you don't know that gets you into trouble, it's what you know that just ain't so!
To get the best posts emailed to you, create an account! (2-3 posts per week, selected by the LessWrong moderation team.)
Log In Reset Password
...or continue with

There's a concept I draw on often in social interactions.  I've been calling it the "happy price", but that is originally terminology by Michael Ellsberg with subtly different implications.  So I now fork off the term "cheerful price", and specialize it anew.  Earlier Facebook discussion here.


  • When I ask you for your Cheerful Price for doing something, I'm asking you for the price that:
    • Gives you a cheerful feeling about the transaction;
    • Makes you feel energized about doing the thing;
    • Doesn't generate an ouch feeling to be associated with me;
    • Means I'm not expending social capital or friendship capital to make the thing happen;
    • Doesn't require the executive part of you, that knows you need money in the long-term, to shout down and override other parts of you.
  • The Cheerful Price is not:
    • A "fair"

Q:  Wait, does that mean that if I give you a Cheerful Price, I'm obligated to accept the same price again in the future?

No, because there may be aversive qualities of a task, or fun qualities of a task, that scale upward or downward with repeating that task.  So the price that makes your inner voices feel cheerful about doing something once, is not necessarily the same price that makes you feel cheerful about doing it twenty times.

I feel like this needs a caveat about plausible deniability. Sometimes the price goes up or down for reasons that I ... (read more)

Education in the US is a big big deal. It takes up 18-30 years of our lives, employs over 10% of our workforce, and is responsible for 60% of non-mortgage/non-car debt. Even a minor improvement to education could be a big deal.

Education is also something that has changed massively in recent decades. In 1930, only 19% of people graduated high school and only 4% went to college1. If something has changed a lot in the past, it is reasonable to expect that it will change a lot in the future. 

And I expect AI to change education a lot.

One-on-one tutoring is known to be far more effective than whole-class teaching2. If someone is listening to a group lecture, half the time they are bored because they are being...

This seems like a great idea -- I haven't tried the GPTs yet, so I will just comment on the rest of the article.

In addition to the AI tutoring the student 1:1, I wonder whether it would be useful to match students with each other based on their current interest and skill level. For humans, it is often motivating to talk to other humans, and the problem is that kids in the same classroom are often not interested in the same topic, or they know much less than you, or they know much more than you so it is boring for them to talk to you. But if this system was... (read more)

Remember, they are not "hallucinations", they are confabulations produced by dream machines i.e. the LLMs!