I've been accepted as a mentor for the next AI Safety Camp. You can apply to work with me and the team.

The deadline for applicants is November 17.

The program will run from January 11 to April 27.

Summary

Core underlying hypothesis - we believe that there is a significant agency overhang in the modern LLMs, meaning there is a potential for performance of a model to increase significantly with introduction of more powerful elicitation/scaffolding methods without additional improvements of model itself, due to prompting and scaffolding techniques being in their early ages. For the model evaluations this means that the current evaluations systematically undershoot the real level of capabilities and by extension, the level of risks involved.

We see several important research questions that have to be answered:

Is the core assumption even true? We want to prove that one can elicit the peak performance using narrow highly specialised prompts and scaffoldings and locally beat general state-of-the-art performance
How overhang should be factored in in the overall model evaluation procedure?
Is it possible to estimate the real level of overhang (e.g. developing an evaluation technique measuring the gap between current sota performance and theoretically possible peak performance)
How big of an increase has been introduced with existing scaffolding techniques?

We are going to decide on which exact paths to pursue later. We expect to remain flexible and shift between paths when presented with new evidence.

The non-summary

This vector of research has been born as an attempt at tackling the problem known as Sharp left turn.

Sharp Left Turn (SLT) is a highly probable situation that can occur in LLMs when the growth of generalisation ability outpaces the growth of alignment measures which leads to those measures rendered ineffectual which in turn may lead to catastrophic consequences.

Assuming we are going to continue with the transformer + scalable oversight + RLHF paradigm you can imagine SLT as follows. There is a state graph of a model. Via fine tuning we prune out the paths leading towards dangerous states. The generalisation here can be viewed as an increase in the number of paths between any two nodes. In this sense SLT might be viewed as an inability to identify and prune out the new dangerous path at the same rate as they are being introduced.

Thi...

Posts

Wikitag Contributions

Comments