Task: Spotting hidden errors and implications in past writings on alignment

• Context: The alignment community has produced a large collection of writing on the alignment problem. This work contains many assumptions, arguments and results that form the basis of our current efforts. If any of these background beliefs are wrong, it could significantly change the structure of the alignment problem. This task would involve teaching an AI to highlight shaky inferences, questionable assumptions and other potential holes or hidden implications in a given line of reasoning. Alignment researchers could then use such AIs to repeatedly scan all past work on the alignment problem, so as to more quickly highlight areas where we could have made a mistake or missed something important.
• Input type: A text containing alignment-relevant reasoning.
• Output type: Text that describes a potential mistake, missing implication or other issue with the alignment-related text in question.
• Additional: These examples just show the key sections of text where the potential mistake / hidden implication occurs. In actual training and execution, the AI would receive the entire alignment-related text (or a large section thereof) as input, and its output would also specify the location in the input text where the issue occurs.

Instance 1:

Input: 

We must take care to avoid inner alignment failures. They can lead to a system with goals that diverge from the base objective to an arbitrarily large degree. Consider that humans are not aligned with evolution. We do things like wear condoms and otherwise fail to maximize our inclusive genetic fitness. Our values diverge arbitrarily widely from those of evolution. 

Output:

Hidden implication: If human values derive from an inner alignment failure, then attempting to prevent inner alignment failures in AI systems means preventing the processes by which human values arose in humans.

Instance 2:

Input:

(From Eliezer Yudkowsky's post My Childhood Role Model)

Humans are adapted to chase deer across the savanna, throw spears into them, cook them, and then—this is probably the part that takes most of the brains—cleverly argue that they deserve to receive a larger share of the meat.

It's amazing that Albert Einstein managed to repurpose a brain like that for the task of doing physics.  This deserves applause.  It deserves more than applause, it deserves a place in the Guinness Book of Records.  Like successfully building the fastest car ever to be made entirely out of Jello.

Output:

Retrodictive error: thinking that the human learning process is highly specialized towards the ancestral environment implies humans would not be able to generalize well beyond the ancestral environment, which does not conform to observed reality.

I think this task will be very difficult for language models to do. I think even Chinchilla may not be quite good enough to be truly useful here. However, I think this task is significantly less difficult than directly making original progress on the core of alignment. 

I also think a lot of useful alignment research is blocked by subtle background assumptions that we don't realize we should question. I basically consider this task to be automating the search for the sort of "miracle" that Yudkowsky sometimes describes as "[violating] some aspect of [his] background model" (source).

This task is also unusual in that even a single true success from such an approach could be enough to entirely change the game in regards to alignment. It would be worth trawling through thousands of false positives to find a single such true positive. The median result of running the system could be garbage. As long as there are occasional gems, it would be worthwhile. Note that large language models do tend to occasionally produce exceptional outputs, even in scenarios where they usually do poorly.

Make it as easy as possible to generate alignment forum posts and comments.

The rough idea here is that it's much easier to explain an idea out loud, especially to someone who occasionally asks for clarification or for you to repeat an idea, than it is to write a clear, concise post on it. Most of the design of this would be small bits of frontend engineering, but language model capability would be useful, and several of the capabilities are things that Ought is already working on. Ideally, interacting with the tool looks like:

Researcher talks through the thing they're thinking about. Model transcribes ideas[1], suggests splits into paragraphs[2], suggests section headings [3], generates a high level summary/abstract [4]. If researcher says "[name of model] I'm stuck", the response is "What are you stuck on?", and simple replies/suggestions are generated by something like this[5].

Once the researcher has talked through the ideas, they are presented with a piece which contains an abstract at the top, then a series of headed sections, each with paragraphs which rather than containing what they said at that point verbatim, contain clear and concise summaries[6] of what was actually said. Clicking on any generated heading allows the user to select from a list of generated alternatives, or write their own[7], while clicking on any paragraph allows the user to see and select from a list of other generated summaries, the verbatim transcription, and to write their own version of this paragraph.

1 can probably be achieved by just buying an off the shelf transcription bot (though you could train one if you wanted), with the most important criterion being speed. 2-4 can have data trivially generated by scraping the entire alignment forum and removing headings/summaries/abstracts/paragraph breaks. 5 I've generated data for below. An MVP for generating data for 6 is using the transcription software from 1 to autotranscribe AXRP and then comparing to the human-edited summary, though I think suggesting clear rephrasings (which I'll call 6.5) might require a seperate task. 7 is just frontend design, which I suspect is doable in-house by Ought.

Task: Suggest surprising experiments that challenge assumptions

Context: A researcher is considering an alignment proposal that hinges on some key assumptions. They would like to see some suggestions for experiments (either theoreetical thoughts experiments or actual real-world experiments) that could challenge those assumptions. If the experiment has been done, it should report the results.

Input type: An assumption about a powerful AI system

Output type: a suggestion for an experiment that could challenge that assumption. If it has been done already, the results of those experiments.

Instance 1:

Input: The performance of a model is impossible to predict, so we can’t hope to have an idea of a model’s capabilities before it is trained and evaluated.

Output: It might be that a key measure of performance of a model, such as the loss, might scale predictably with the model size. This was investigated by Kaplan et al (https://arxiv.org/abs/2001.08361), who found that the loss tends to follow a power law.

Instance 2:

Input: Suppose a model is trained on data that is mixed with some noise (as in https://arxiv.org/pdf/2009.08092.pdf ).The model will necessarily learn that the data was mixed with some noise, rather than learn a really complex decision boundary.

Output: Suppose that you try fine-tuning one of these models on data that doesn’t have the noise. It might be very slow to adapt to this in which case it might have learned the complex decision boundary. (This experiment hasn’t been done.)

Instance 3:

Input: It's impossible to train a neural network without non-linearities like ReLU or a sigmoid.

Output: That is true for theoretical neural networks, but real neural networks are trained using floating point numbers with inherently non-linear arithmetic. These imperfections might be enough to train a competent model. This experiment was done by Jakob Foerster, who found that this was indeed enough: https://openai.com/blog/nonlinear-computation-in-linear-networks/

Task: Apply an abstract proposal to a concrete ML system

Context: A researcher is reading a highly theoretical alignment paper and is curious about whether/how it might apply to a real world machine learning system, like a large transformer trained using SGD. They would like to see what parts of the ML system would change under this proposal.

Input type: a theoretical alignment proposal and a description of an ML system

Output type: a description of how the ML system would change under the given proposal

Info constraints: none

Instance 1:

Input:

Abstract proposal: The description of the complexity regularizer from the ELK report:

ML system description: a GPT-style model trained on natural language using SGD, with a GPT-style reporter trained on the GPT model’s weights.

Output: This is tricky becuase the circuit complexity of a neural network is largely fixed by its architecture. As we sweep over different architecutres/hyperparemeters for the reporter model, we can add a regularization term to the hyperparameter optimization based on the total number of weights in the model.

Other possible output: Since circuit complexity for a neural network is largely fixed by its architecture, we will consider the “minimal complexity” of the trained neural network to be the number of non-zero parameters, and will regularize on that to encourage sparsity in the weights.

Instance 2:

Input:

Abstract proposal: The Iterated Amplification and Distillation proposal, as described in https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616

ML system description: a GPT-style model trained on natural language using SGD.

Output: Any of the project proposals in https://www.alignmentforum.org/posts/Y9xD78kufNsF7wL6f/machine-learning-projects-on-ida, or something similar

Interactive directory of alignment researchers, organizations, and funding bodies

Context: Connecting people who want to work on or fund alignment research with the right collaborators is a high leverage activity, but as the field grows methods conducting searches via Google and LinkedIn will take a lot of time and won’t always produce comprehensive results. This system would be useful if it could let its user ask who is working on a particular type of project and get the same answer you’d get if you asked someone well-connected and up-to-date on the research in the relevant subject area.

Input Type:  A question about the people, organizations, or funding bodies in a particular subfield of alignment research. 

Output Type:  A list of the relevant entities and a brief explanation of why they are relevant. If possible, it would be helpful to also provide contact information.

Instance 1: 

Input

Who is working on using adversarial examples to make models more robust?

Output

Redwood Research’s current project uses adversarial examples.

The FTX Future Fund is also interested in funding related projects, including the Unrestricted Advex Challenge and achieving near-perfect robustness on adversarial examples in vision. 

Instance 2:

Input

Who is working on fine-tuning large language models to be more aligned with human instructions?

Output

Long Ouyang, Jeff Wu, and others are working on this at OpenAI (https://arxiv.org/abs/2203.02155).

Yuntao Bai, Andy Bai, Kamal Ndousse, and others are also working on this problem at Anthropic (https://arxiv.org/abs/2204.05862).

Instance 3:

Input

Who has experience creating interactive visualizations to help understand transformer models?

Output

Ben Hoover, Hendrik Strobel, and Sebastian Gehrmann worked on this with the exBERT project.
Chris Olah is working on similar projects related to Transformer Circuits.

Instance 4:

Input

What organizations are funding the creation of open datasets for alignment research?

Output

MIRI has a one million dollar bounty for the Visible Thoughts Project.

Identify the novel ideas in a paper

• Context: An alignment researcher has a database of notes on research papers. They encounter a new paper, and would like to know which ideas in that paper are new to them. A system is useful if it can summarize just the claims/ideas in a paper which do not appear in the researcher’s database.
• Input type: A database of notes on research papers structured as (paper, notes) tuples, along with a new research paper.
• Output type: A list of new ideas/claims in the paper, along with pointers to where they appear in the text.
• Info constraints: None

The instances below are a bit sparse because each instance required summarizing at least some ideas from multiple posts. Fortunately instances “nest”. As a researcher reads new posts, each new post generates a new instance (with all previous posts forming the database).

Instance 1:

Input:

Database: (Musings on the Speed Prior: Deception takes longer than accomplishing the original goal, so the speed prior favors just doing what the human wants. The speed prior may perform badly on many tasks though, so there is a competitiveness concern.)

New Paper: Should we rely on the speed prior for safety?

Output:

New idea(s): A speed prior could disfavor meta-learning over directly learning the correct policy.

Instance 2:

Database: ((Musings on the Speed Prior: Deception takes longer than accomplishing the original goal, so the speed prior favors just doing what the human wants. The speed prior may perform badly on many tasks though, so there is a competitiveness concern.), (Should we rely on the speed prior for safety?: A speed prior could disfavor meta-learning over directly learning the correct policy.))

New Paper: The Speed+Simplicity Prior is probably anti-deceptive

Output:

New idea(s): Early deception requires a more complex and time-consuming program than corrigibility, so speed and simplicity priors bias towards corrigibility. Corrigibility requires a more complex program but less runtime than late deception, so speed priors push towards corrigibility while simplicity priors push towards late deception.

Summarize an Alignment Proposal in the form of a Training Story

• Context: An alignment researcher has written a proposal for building a safe AI system. To better understand the core of the idea, they want to summarize their proposal in the form of a training story.
• Input type: A proposal for building a safe AI system.
• Output type: A summary of the proposal in the form of a training story.
• Info constraints: None.

Both instances are taken directly from How do we become confident in the safety of a machine learning system?

Instance 1:

• Input: Paul Christiano’s description of corrigibility.
• Output: Paul Christiano’s description of the “intended model”.

Instance 2:

• Input: Chris Olah’s Microscope AI proposal.
• Output: (verbatim from How do we become confident in the safety of a machine learning system?)

The training goal of Microscope AI is a purely predictive model that internally makes use of human-understandable concepts to be able to predict the data given to it, without reasoning about the effects of its predictions on the world. Thus, we can think of Microscope AI’s training goal as having two key components:

1. the model doesn’t try to optimize anything over the world, instead being composed solely of a world model and a pure predictor; and
2. the model uses human-understandable concepts to do so.

The reason that we want such a model is so that we can do transparency and interpretability on it, which should hopefully allow us to extract the human-understandable concepts learned by the model. Then, the idea is that this will be useful because we can use those concepts to help improve human understanding and decision-making.

The plan for getting there is to do self-supervised learning on a large, diverse dataset while using transparency tools during training to check that the correct training goal is being learned. Primarily, the training rationale is to use the nudge of an inductive bias towards simplicity to ensure that we get the desired training goal. This relies on it being the case that the simplest algorithm that’s implementable on a large neural network and successfully predicts the training data is a straightforward/pure predictor—and one that uses human-understandable concepts to do so. The use of transparency tools during training is then mostly just to verify that such a nudge is in fact sufficient, helping to catch the presence of any sort of agentic optimization so that training can be halted in such a case.

Merge Synonyms in Draft Research Papers

• Context: An alignment researcher is trying to draft a research paper, but uses multiple words for the same ideas, causing confusion in the reader. The system is useful if it can identify when this occurs and suggest terminology to standardize around. The system is especially useful if the suggested terminology matches what is used in any papers the draft refers to.
• Input type: A draft of a research paper.
• Output type: A list of possibly synonymous concepts appearing in the paper, and suggestion for standardized terminology.
• Info Constraints: None.

Instance 1:

• Input: Say you want to train an agent to act in an environment and optimize some goal. In the language of inner alignment, the goal being optimized is the base objective. The model is going to end up with a policy. That policy may not directly optimize the base objective, but instead targets a mesa objective.
• Output: The draft uses “agent” and “model” interchangeably. It would be clearer to standardize around “model”, because that is what the linked paper uses.

Instance 2:

• Input: ELK Proposal: Thinking Via a Human Imitator
• Output: The terms “AI” and “agent” may be synonyms in this proposal. Consider standardizing around one or the other. It is not clear which to standardize around, because the ELK report uses both terms.

Task: Check understanding of a paper.

• Context: An alignment researcher has read a research paper, but is unsure of if they understand its core ideas. The system is useful for them if it helps them check their understanding of the paper’s claims.
• Input type: Research paper and a summary of some of the paper’s claims written by the researcher.
• Output type: Correction to the researcher’s summary, or confirmation that the researcher has produced a valid summary.
• Info constraints: None.

Instance 1:

• Input:
  • Source: ARC’s first technical report
  • Researcher: Comment summarizing some aspects of the report.
• Output:
  • Reply clarifying some aspects of the summary.

Instance 2:

• Input:
  • Source: Introduction to Cartesian Frames
  • Researcher: Summary for the Alignment Newsletter
• Output:
  • Confirmation that this is a good summary.

(I think this task scales well because LW has lots of examples already)

Task: Expand on a poorly-defined research idea

Context: Often, people have vague intuitions about what to research, but find it hard to flesh out that idea or are unable to communicate it to others. By clarifying an under-specified thought, researchers can quickly identify fruitful directions for research.

Input type: One sentence that vaguely defines an idea.

Output type: 1 to 3 paragraphs providing a clearer, more detailed explanation of the idea and some guidance on where to research further.

Instance 1

Input: The proliferation of drones as weapons seems bad.

Output: In Ukraine, bombs dropped from small drones have been used to destroy tanks and target military personnel. Some experts are worried that drones could be used to carry out assassinations or terrorist attacks. Drones are difficult to defend against and the fact that missions can be completed autonomously makes it hard to retaliate against the perpetrator. It is important to develop technology that can identify, track, and disable many malicious drones simultaneously.

Instance 2

Input: Immigration policy is relevant for international relations.

Output: Immigration policy can be a useful tool to improve international relations. For one, increased immigration between two countries can improve diplomacy by fostering connections between citizens and by increasing cultural awareness. Additionally, easy migration allows people to move away from oppressive regimes, incentivizing governments to treat citizens well in order to maintain their power. Research on how exposure to other cultures increases cooperation across groups would be valuable. Additionally, policies to increase global mobility can weaken oppressive regimes.

Task: Computing the relevance of a paper to solving a problem

Context: A researcher is looking at existing literature trying to solve some problem. However, it is not obvious what to search for. There might exist research whose relevance can only be determined once they read it and think about it. For this task a simple keyword-matching search engine isn’t useful.

Input type: A problem statement and an URL to a website or PDF. Optionally, text with reasoning about the problem and what has been tried before. The URLs in the training set can be papers that have been found useful in the past, but ideally they would be found by interacting with the system. When queried it evaluates as many pages in its crawling corpus as possible, perhaps only the top k (for a very big k) results given by a regular search engine.

Output type: An explanation (maybe optional, considering most pages are irrelevant) of why the document might or might not be useful for solving the problem, in addition to a numerical score. The score can be given in some scale of usefulness following some consistent guidelines. But ideally it would be the percentile of usefulness assuming it falls among the n pages the researcher can look at when interacting with the system.

Info constraints: If training with documents that have been useful in the past, the program can’t be aware of any work that has been the result of that.

Instances: (they should be collected by interacting with the system, the usefulness percentiles are made up)

• Instance 1:
  • Input: 
    • Problem: Ontological translation. Given two different ontologies about the same domain and some representation in one ontology, compute the corresponding representation (if any) on the other ontology.
    • URL: Ontological Crisis in Artificial Agents’ Value Systems
  • Output:
    • Reasoning: This paper deals with translating between an AI’s old wrong ontology, represented as a POMDP-like system and a new bigger and better ontology. It presents a simple heuristic algorithm for doing so based on bisimulation. It has shortcomings, such as using monolithic states or being plausibly susceptible to steganography, but it seems like a promising starting point.
    • Predicted usefulness percentile: 80%
• Instance 2:
  • Input: 
    • Problem: Ontological translation.
    • URL: An ontological approach to translation and untranslatability
  • Output:
    • Reasoning: This is a paper about linguistics that deals with language translation. It contains no insights about general ontological translation.
    • Predicted usefulness percentile: <1%
• Instance 3:
  • Input: 
    • Problem: Ontological translation.
    • URL: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
  • Output:
    • Reasoning: This paper describes a method for automatically computing an image in some distribution corresponding to a given image in another distribution, say: photos of apples to photos of oranges or zebras to horses. Cycle consistency might be useful in general. However, the loss:
      1. Doesn’t guarantee that we will actually end up with the corresponding element, only something that seems to be from that distribution. While it works in this case, it is probably because doing it correctly happens to be simple, there is no guarantee that it will generalize to more complex situations.
      2. Seems unusually susceptible to steganography.
      3. Has the regular problems associated with GANs.
    • Predicted usefulness percentile: 63%
• Instance 4:
  • Input: 
    • Problem: Ontological translation.
    • URL: Unsupervised Machine Translation Using Monolingual Corpora Only
  • Output:
    • Reasoning: This paper presents an algorithm for translation sentences between two languages in an unsupervised fashion. It is based on encoding sentences of both languages in the same latent space, forcing those two distributions to be indistinguishable using an adversarial loss and useful for performing two different tasks. While it works for this use case, it uses text translation specific tricks such as initializing the translator to word-by-word translation obtained using another method or adding “noise” by dropping and swapping words. It also doesn't seem like it could scale well, especially if the sizes of the ontologies are very different.
    • Predicted usefulness percentile: 45%

Simply searching for “ontological translation” on Google Scholar gives terrible results.

I might add more and better instances later.

Draw a diagram of a model from a description

• Context: An alignment researcher has written a description of a model they intend to build, and would like a diagram of that model. A system is useful if it can draw a publication-ready diagram of the model.
• Input type: A description of a model.
• Output type: A diagram of that model. This could be in an image file format or in a diagramming language like Mermaid.
• Info constraints: None.

There are lots of examples of this kind of task from existing papers/posts, but it may take some curation to pull all the relevant context into the descriptions (e.g. it might be interspersed with text on other topics).

Instance 1:

Input: The section “Baseline: what you’d try first and how it could fail” from the ELK report.

Output: This diagram from the ELK report:

Instance 2:

Input: A model is trained on inputs $x$ and outputs $y$. The outputs are generated by computing a simple function $f\left(x\right)$ and adding some noise. The model learns the map from $x$ to $y$ by learning separately the function $f\left(x\right)$ and a memorized table of the noise.

Output:

or as Mermaid code:

graph LR;
	x --> f;
  x --> Noise;
  f --> +;
	Noise --> +;
	+ --> Output;

Instance 3: (From A Mathematical Framework for Transformer Circuits)

Input:

There are several variants of transformer language models. We focus on autoregressive, decoder-only transformer language models, such as GPT-3. (The original transformer paper had a special encoder-decoder structure to support translation, but many modern language models don't include this.)

A transformer starts with a token embedding, followed by a series of “residual blocks”, and finally a token unembedding. Each residual block consists of an attention layer, followed by an MLP layer. Both the attention and MLP layers each “read” their input from the residual stream (by performing a linear projection), and then “write” their result to the residual stream by adding a linear projection back in. Each attention layer consists of multiple heads, which operate in parallel.

Output: