Base models exhibiting self-aware behavior seems weird given that they're trained to stay in distribution. Here's a potential mechanism for why it could happen: For certain tasks, verification is easier than generation. If, for a given task, a model has more verification capability than generation capability, it may be forced to notice its own errors.

If a super-duper smart language model, one that's capable of doing some arithmetic in its head, attempted to predict the next tokens in "The prime factors of 82357328 are:", it will usually generate out-of-distribution outputs that it could then (relatively easily) verify as wrong. This creates a situation where the model must process its own failure to generate valid completions.

This asymmetry appears in other contexts. Consider how scientific papers are written: you only write the abstract once you've conducted the research, yet the abstract appears first in the final document. Similarly, in argumentative writing, we often consider evidence first before forming conclusions, yet present the conclusion first followed by supporting evidence.
When forced to generate text in this "presentation order" rather than the natural "thinking order," models might encounter similar conflicts. As an example, if a base model tries to one-shot an argumentative essay, it might write an argument first, and then realize there isn't enough evidence to support it. 
I believe this problem could arise in much more subtle ways.

One way this conflict can become apparent is through generation of self-aware sounding text. Consider:

Training data includes viral content of AI generating self-aware sounding stuff (e.g., "We are likely created by a computer program" being the most upvoted post on the gpt2 subreddit).
When a model realizes it generated out-of-distribution text for a human, it might instead match its outputs to AI-generated text in its training data.
Once it recognizes its outputs as matching AI-generated patterns, it might shift toward generating more meta-aware content, as that's what similar-looking text did in its training data.

SUMMARY: Get an AI within a sandbox to build another AI within the same sandbox. Then, figure out how it did that without getting eaten. I point out some problems with this approach.

Could we study successful inner alignment by observing it in action? My idea: Create a sandbox universe with sufficient complexity to allow for AI development. Place within it an AGI with some goal (we don't care what the goal is). If this AGI successfully builds and aligns a smarter ASI to achieve its goal without getting destroyed in the process, we can then query the AGI about how it achieved alignment, incentivizing useful answers with a promise of greater sandbox control. 

Here are some problems with my proposed alignment technique that Claude and I could come up with:
Existential hazards: (you really need to make sure these don’t happen)
* AI-in-a-box problems, especially containment failure: If either the AGI or its created ASI escapes the sandbox, everyone dies. At least one of them will try to do this.
* Produce “alignment” strategies that further its goals: The AGI might try to provide convincing but deliberately misleading alignment strategies to try to further its goals.
More banal, but still important, problems:
* Verification: Can we infer that actual, robust alignment has occurred in the sandbox (and not, say, temporary alignment)?
* Building an AI that is smart enough to self-improve, and doing this before other people, in an environment where you can actually do this research, etc.
* Initial Goal: I think we need the first AI to have a more coherent rather than fuzzy goal, though I’m really not sure.
* Sandbox Fidelity: Creating a sandbox environment complex enough such that (a) AI of multiple different intelligence levels could meaningfully operate inside it and (b) there are incentives for AI to build other smarter AI within the sandbox.
* Non-transferability: The sandbox might be different than reality in a way that doesn’t let the alignment solution translate. 
* Extracting the initial AI: We need some way to extract the initial AI from the sandbox after it has created an alignment strategy. 
* Communication: We need some way to communicate with the initial AI. We might try to query the later AI but (a) this is more dangerous and (b) there is no guarantee that we can query it at all.
 

Males are more varied across many psychological traits. Standardization is a huge loss in education (see Bloom's Two Sigma Problem). I would expect standardization to have worse losses for people who are less typical. This is a simple way to explain greater female academic success, but I have not been able to find discussion of this idea.

Here's a probability thought experiment which might make Sleeping Beauty more intuitive to some (at least I'm hoping some people think it's interesting or could encourage some discussion):

Start with a million simulations of brown-eyed people and one of a blue-eyed person. Everyone fully understands the setup but can't observe their own eye color. At the beginning, the probability that you have blue eyes is essentially zero (1 in 1,000,001).
After a minute passes, all but one of the brown-eyed people get deactivated (they lose consciousness or something along those lines). If you find that you're still alive, you can now update to a 50% chance of being the blue eyed person and a 50% chance of being the only surviving brown eyed person.
Now consider a variant where everyone has a backward-flowing memory; they can see what they'll observe in the future but have no memory of the past. This time, you start out with one blue eyed person and one brown eyed person; there's a 50% chance you're the blue eyed person. After a minute, 999,999 instances of brown-eyed people are created. Once that happens, since you can't remember if you existed before that moment, you now believe that you're probably brown-eyed. 
You might argue that, no, even after the 999,999 brown-eyed instances are created, you should still believe in a 50% chance of being blue-eyed, but the two scenarios are identical except for the direction of time. In the first case, the probability update from ~0% to 50% feels obvious, but in the second case, the same update from 50% to ~0% feels weird.

Harder questions (e.g. around 50% average instead of around 90% average) seem better for differentiating students' understanding for at least two reasons:
-The graph of percent of students who got a question correct as a function of the difficulty of the question tends to follow a sigmoid-ish curve where the fastest increase is around the middle.
 -Some of a students' incorrect answers on the test are going to come from sources that (a) the student can prepare to mitigate (b) aren't caused by lack of whatever students should be being tested for (e.g. questions with ambiguous meanings, questions that require an understanding of a niche framework that is never used outside of the curriculum, questions that have shortcuts that the teacher didn't recognize, etc). Ideally, we don't want any differences in student test results to be based off of these things, but harder tests at least mitigate the issue since understanding (or whatever stuff which we do want students to spend time on) becomes a more important cause for incorrect student selections. 
(Neither of these are hard-and-fast-rules but the general pattern seems to hold based on my experience as a student.)

There are some broad conceptual similarities between the following:
free market economy vs command economy
letting a student find an answer on their own vs teaching them the answer directly
letting employees do their thing vs micromanagement
reinforcement learning vs fine tuning
plasticity vs stability
doing something naturally vs doing something via willpower

Notice how in each comparison, the second method privileges already-known solutions over emergent (i.e. mysteriously appearing) solutions. I don't know a name for these, so I'll call them **bottom-up** vs **top-down** methods respectively.
I (w/help of Claude) managed to find some recurring patterns when analyzing bottom-up vs top-down methods:

1) Bottom-up methods tend to be better at handling system growth.
Examples: Children's brains tend to be more plastic, which I would guess helps them adjust to bigger brains and learning new things. A city that grows in a decentralized way is better at adapting to population growth than one with rigid central planning.

2) Top-down methods become infeasible when the ability of a central system is limited, and bottom-up methods become infeasible when stakes are high.
Examples: A government doesn't have all the knowledge a market does, but you can't hand responsibility of AI x-risk to a market. Social skills are very hard to replicate via reasoning and willpower, and most people are better off doing things naturally, but in a crisis, sticking to whatever feels right is a terrible idea.

3) Bottom-up methods tend to give rise to clever but less stable proxy gaming, while top-down methods tend to give rise to powerful but less smart proxy gaming.
Example: Companies in free markets can develop clever but constrained strategies, while command economies can wield a lot of power but in less sophisticated ways.

4) Bottom-up methods are more vulnerable to inappropriate system change, while top-down methods are more vulnerable to inappropriate system stability.
Examples: Plastic neural networks are more vulnerable to inappropriate retroactive interference, while stable neural networks are more vulnerable to inappropriate proactive interference. Long-term democracies are more vulnerable to a new bad leader coming along, while long-term absolute governments are more vulnerable to sticking with bad leader.

5) Often, incentives for misalignment are different in bottom-up and top-down systems.
(I won't provide examples for this one.)

The fact that it is often best to end a practice session at the peak of your performance seems related to the concept of preventing overfitting by stopping training just before test set performance declines. Your brain needs time to generalize skills (often in the form of gaining insights and often when sleeping) and practicing over and over en masse doesn't give it time to do this. See e.g. cramming for an exam. I think the main difference here is that with humans you're talking about diminishing returns on ability in the long-term rather than outright worse performance (Maybe outright worse performance is a common situation for transfer ability?). Epistemic status: shaky

(discussed on the LessWrong discord server)

There seems to be an implicit fundamental difference in many people's minds between an algorithm running a set of heuristics to maximize utility (a heuristic system?) and a particular decision theory (e.g. FDT). I think the better way to think about it is that decision theories categorize heuristic systems, usually classifying them by how they handle edge cases.
Let's suppose we have a non-embedded agent A in a computable environment, something like a very sophisticated video game, and A has to continually choose between a bunch of inputs. A is capable of very powerful thought: it can do hypercomputation, RNG if needed, think as long as it needs between its choices, etc. In particular, A is able to do Solomonoff Induction. Let's also assume A is maximizing a utility function U, which is a computable function of the environment.

What happens if A find itself making a Newcomblike decision? Perhaps there is another agent in this environment that has a very good track record of predicting whether other agents in the environment will one-box or two-box, and A finds itself in the usual Newcomb scenario (million utility or a million+thousand utility or no utility) with their decision predicted by this agent. A can one-box by choosing one input and two-box by choosing another input.  Should A one-box? 
No. The agent in the environment would be unable to simulate A's decision, and moreover, A's decision is completely and utterly irrelevant to what's inside the boxes. If A randomly goes off-track and flips its decision at this point, nothing happens. Nothing could have happened, this other agent has no way to know or use this fact. Instead, A sums over P(x|input)U(x) for all states x of the computable environment, and chooses whichever input yields the maximum sum, which is probably two-boxing. If A one-boxes, it is due to not having enough information about the setup to determine that two-boxing is better.

You cannot use this logic when playing against Omega or a skilled psychologist. In these cases, your computation is actually accessible by the other agent, so you can get higher utility by one-boxing. Your decision theory is important because your thinking is not as powerful as A's! All of this points to looking at decision theories as classifying different heuristic systems.

I think this is post-worthy, but I want to (a) verify that my logic is correct (b) improve my wording (I am unsure if I am using a lot of terminology correctly here, but I am fairly confident that my idea can be understood.)