My hunch is that there's sufficient text already if an AI processes it more reflectively. For example, each chunk of text can be fed through a series of LLM prompts intended to enrich it, and then the model trains on the enriched/expanded text.
I definitely like the term "confabulate" more than "hallucinate". It's more accurate and similar to what humans do. My favorite confabulation examples in humans are split-brain experiments.
"split-brain" patients, whose left and right brain hemispheres have been surgically disconnected for medical treatment. Neuroscientists have devised clever experiments in which information is provided to the right hemisphere (for instance, pictures of naked people), causing a change in behavior (embarrassed giggling). Split-brain individuals are then asked to explain their behavior verbally, which relies on the left hemisphere. Realizing that their body is laughing, but unaware of the nude images, the left hemisphere will confabulate an excuse for the body's behavior ("I keep laughing because you ask such funny questions, Doc!").
I found your thread insightful, so I hope you don't mind me pasting it below to make it easier for other readers.
Neel Nanda ✅ @NeelNanda5 - Sep 24
The core intuition is that "When you see 'A is', output B" is implemented as an asymmetric look-up table, with an entry for A->B. B->A would be a separate entry
The key question to ask with a mystery like this about models is what algorithms are needed to get the correct answer, and how these can be implemented in transformer weights. These are what get reinforced when fine-tuning.
The two hard parts of "A is B" are recognising the input tokens A (out of all possible input tokens) and connecting this to the action to output tokens B (out of all possible output tokens). These are both hard! Further, the A -> B look-up must happen on a single token position
Intuitively, the algorithm here has early attention heads attend to the prev token to create a previous token subspace on the Cruise token. Then an MLP neuron activates on "Current==Cruise & Prev==Tom" and outputs "Output=Mary", "Next Output=Lee" and "Next Next Output=Pfeiffer"
"Output=Mary" directly connects to the unembed, and "Next Output=Lee" etc gets moved by late attention heads to subsequent tokens once Mary is output.
Crucially, there's an asymmetry between "input A" and "output A". Inputs are around at early layers, come from input embeddings, and touch the input weights of MLP neurons. Outputs are around more at late layers, compose with the unembedding, and come from output weights of MLPs
This is especially true with multi-token A and B. Detecting "Tom Cruise" is saying "the current token embedding is Cruise, and the prev token space says Tom", while output "Tom Cruise" means to output the token Tom, and then a late attn head move "output Cruise" to the next token
Thus, when given a gradient signal to output B given "A is" it reinforces/creates a lookup "A -> B", but doesn't create "B->A", these are different lookups, in different parameters, and there's no gradient signal from one to the other.
How can you fix this? Honestly, I can't think of anything. I broadly think of this as LLMs working as intended. They have a 1 way flow from inputs to outputs, and a fundamental asymmetry between inputs and outputs. It's wild to me to expect symmetry/flow reversing to be possible
Why is this surprising at all then? My guess is that symmetry is intuitive to us, and we're used to LLMs being capable of surprising and impressive things, so it's weird to see something seemingly basic missing.
LLMs are not human! Certain things are easy for us and not for them, and vice versa. My guess is that the key difference here is that when detecting/outputting specific tokens, the LLM has no notion of a variable that can take on arbitrary values - a direction has fixed meaning
A better analogy might be in-context learning, where LLMs CAN use "variables". The text "Tom Cruise is the son of Mary Lee Pfeiffer. Mary Lee Pfeiffer is the mother of..." has the algorithmic solution "Attend to the subject of sentence 1 (Tom Cruise), and copy to the output"
Unsurprisingly, the model has no issue with reversing facts in context! Intuitively, when I remember a fact A is B, it's closer to a mix of retrieving it into my "context window" and then doing in-context learning, rather than pure memorised recall.
Adversarial gaming doesn't match my experience much at all and suggesting options doesn't feel imposing either. For me at least, it's largely about the responsibility and mental exertion of planning.
In my experience, mutual "where do you want to go" is most often when neither party has a strong preference and neither feels like taking on the cognitive burden of weighing options to come to a decision. Making decisions takes effort especially when there isn't a clearly articulated set of options and tradeoffs to consider.
For practical purposes, one person should provide 2-4 options they're OK with and the other person can pick one option or veto some option(s). If they veto all given options, they must provide their own set of options the first person can choose or veto. Repeat as needed but rarely is more than one round needed unless participants are picky or disagreeable.
Why do you think that the same federal bureaucrats who incompetently overregulate other industries will do a better job regulating AI?
Chevron deference means that judges defer to federal agencies instead of interpreting the laws themselves where the statute is ambiguous. It's not so much a question of overregulation vs underregulation as it is about who is doing the interpretation. For example, would you rather the career bureaucrats in the Environmental Protection Agency determine what regulations are appropriate to protect drinking water or random judges without any relevant expertise?
One consequence of blowing up Chevron deference is that one activist judge in Texas can unilaterally invalidate FDA approval of a drug like mifepristone for the entire country that's been safe, effective, and available on the markets for decades by substituting his own idiosyncratic opinion instead of deferring to the regulatory agency whose entire purpose is to make those kinds of determinations.
Government agencies aren't always competent but the alternative is a patchwork of potentially conflicting decisions from judges ruling outside of their area of expertise.
If you're open to more experimentation, I'd recommend trying playing against Leela Chess Zero using some of the newer contempt parameters introduced in this PR and available in the latest pre-release version. I'm really curious if you'd notice significant style differences with different contempt settings.
Update: The official v0.30.0 release is out now and there is a blog post detailing the contempt settings. Additionally, there is a Lichess bot set up specifically for knight odds games.
"Artificial Intention" doesn't sound catchy at all to me, but that's just my opinion.
Personally, I prefer to think of the "Alignment Problem" more generally rather than "AI Alignment". Regardless of who has the most power (humans, AI, cyborgs, aliens, etc.) and who has superior ethics, conflict arises when participants in a system are not all aligned.
This is the most similar question that I could find that already exists.
https://www.metaculus.com/questions/7384/alien-tech-in-solar-system-before-2030/
There are still HR and legal overhead costs involved if you have human operators.
I think part of the answer is also that the space of things low-wage workers can physically do remotely via teleoperation isn't that much larger than the space of things that can be fully automated but still much smaller than the space of things a local human can do. It's a fairly narrow band to exploit, IMO, and the labor cost arbitrage spread is rarely worth the complexity of the extra logistics, capital investment, and maintenance.
That sounds rather tautological.
Assuming ratfic represents LessWrong-style rationality well and assuming LW-style rationality is a good approximation of truly useful instrumental reasoning, then the claim should hold. There’s room for error in both assumptions.