LESSWRONG
LW

Stephen McAleese
1171Ω29111901
Message
Dialogue
Subscribe

Software Engineer interested in AI and AI safety.

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
3Stephen McAleese's Shortform
2y
13
Foom & Doom 2: Technical alignment is hard
Stephen McAleese2d50

Thank you for the reply!

Ok but I still feel somewhat more optimistic about reward learning working. Here are some reasons:

  • It's often the case that evaluation is easier than generation which would give the classifier an edge over the generator.
  • It's possible to make the classifier just as smart as the generator: this is already done in RLHF today: the generator is an LLM and the reward model is also based on an LLM.
  • It seems like there are quite a few examples of learned classifiers working well in practice:
    • It's hard to write spam that gets past an email spam classifier.
    • It's hard to jailbreak LLMs.
    • It's hard to write a bad paper that is accepted to a top ML conference or a bad blog post that gets lots of upvotes.

That said, from what I've read, researchers doing RL with verifiable rewards with LLMs (e.g. see the DeepSeek R1 paper) have only had success so far with rule-based rewards rather than learned reward functions. Quote from the DeepSeek R1 paper:

We do not apply the outcome or process neural reward model in developing DeepSeek-R1-Zero, because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process, and retraining the reward model needs additional training resources and it complicates the whole training pipeline.

So I think we'll have to wait and see if people can successfully train LLMs to solve hard problems using learned RL reward functions in a way similar to RL with verifiable rewards.
 

Reply
Foom & Doom 2: Technical alignment is hard
Stephen McAleese3d*50

In the post you say that human programmers will write the AI's reward function and there will be one step of indirection (and that the focus is the outer alignment problem).

But it seems likely to me that programmers won't know what code to write for the reward function since it would be hard to encode complex human values. In Superintelligence, Nick Bostrom calls this manual approach "direct specification" of values and argues that it's naive. Instead, it seems likely to be that programmers will continue to use reward learning algorithms like RLHF where:

  1. The human programmers have a dataset of correct behaviors or a natural language description of what they want and they use this information to create a reward function or model automatically (e.g. Text2Reward).
  2. This learned reward model or generated code is used to train the policy.

If this happens then I think the evolution analogy would apply where there is some outer optimizer like natural selection that is choosing the reward function and then the reward function is the inner objective that is shaping the AI's behavior directly.

Edit: see AGI will have learnt reward functions for an in-depth post on the subject.

Reply
Foom & Doom 1: “Brain in a box in a basement”
Stephen McAleese5d42

I think it depends on the context. It's the norm for employees in companies to have managers though as @Steven Byrnes said, this is partially for motivational purposes since the incentives of employees are often not fully aligned with those of the company. So this example is arguably more of an alignment than a capability problem.

I can think of some other examples of humans acting in highly autonomous ways:

  • To the best of my knowledge, most academics and PhD students are expected to publish novel research in a highly autonomous way.
  • Novelists can work with a lot of autonomy when writing a book (though they're a minority).
  • There are also a lot of personal non-work goals like saving for retirement or raising kids which require high autonomy over a long period of time.
  • Small groups of people like a startup can work autonomously for years without going off the rails like a group of LLMs probably would after a while (e.g. the Claude bliss attractor).
Reply
Foom & Doom 1: “Brain in a box in a basement”
Stephen McAleese5d30

Excellent post, thank you for taking the time to articulate your ideas in a high-quality and detailed way. I think this is a fantastic addition to LessWrong and the Alignment Forum. It offers a novel perspective on AI risk and does so in a curious and truth-seeking manner that's aimed at genuinely understanding different viewpoints.

Here are a few thoughts on the content of the first post:

I like how it offers a radical perspective on AGI in terms of human intelligence and describes the definition in an intuitive way. This is necessary as increasingly AGI is being redefined as something like "whatever LLM comes out next year". I definitely found the post illuminating and resulted in a perspective shift because it described an important but neglected vision of how AGI might develop. It feels like the discourse around LLMs is sucking the oxygen out of the room, making it difficult to seriously consider alternative scenarios.

I think the basic idea in the post is that LLMs are built by applying an increasing amount of compute to transformers trained via self-supervised or imitation learning but LLMs will be replaced by a future brain-like paradigm that will need much less compute while being much more effective.

This is a surprising prediction because it seems to run counter to Rich Sutton's bitter lesson which observes that, historically, general methods that leverage computation (like search and learning) have ultimately proven more effective than those that rely on human-designed cleverness or domain knowledge. The post seems to predict a reversal of this long-standing trend (or I'm just misunderstanding the lesson), where a more complex, insight-driven architecture will win out over simply scaling the current simple ones.

On the other hand, there is an ongoing trend of algorithmic progress and increasing computational efficiency which could smoothly lead to the future described in this post (though the post seems to describe a more discontinuous break between current and future AI paradigms).

If the post's prediction comes true, then I think we might see a new "biological lesson": brain-like algorithms will replace deep learning which replaced GOFAI.

Reply
the void
Stephen McAleese11d00

The post mentions Janus’s “Simulators” LessWrong blog post which was very popular in 2022 and received hundreds of upvotes.

Reply
Mikhail Samin's Shortform
Stephen McAleese25d20

Anthropic’s responsible scaling policy does mention pausing scaling if the capabilities of their models exceeds their best safety methods:

“We have designed the ASL system to strike a balance between effectively targeting catastrophic risk and incentivising beneficial applications and safety progress. On the one hand, the ASL system implicitly requires us to temporarily pause training of more powerful models if our AI scaling outstrips our ability to comply with the necessary safety procedures. But it does so in a way that directly incentivizes us to solve the necessary safety issues as a way to unlock further scaling, and allows us to use the most powerful models from the previous ASL level as a tool for developing safety features for the next level.”

I think OP and others in the thread are wondering why Anthropic doesn’t stop scaling now given the risks. I think the reason why is that in practice doing so would create a lot of problems:

  • How would Anthropic fund their safety research if Claude is no longer SOTA and becomes less popular?
  • Is Anthropic supposed to learn from and test only models at current levels of capability and how does it learn about future advanced model behaviors? I haven’t heard a compelling argument for how we could solve superalignment by studying much less advanced models. Imagine trying to align GPT-4 or o3 by only studying and testing GPT-2 from 2019. In reality, future models will probably have lots of unknown unknowns and emergent properties that are difficult or impossible to predict in advance. And then there’s all the social consequences of AI like misuse which are difficult to predict in advance.

Although I’m skeptical that alignment can be solved without a lot of empirical work on frontier models I still think it would better if AI progress were slower.

Reply
How to work through the ARENA program on your own
Stephen McAleese1mo50

Thanks for the guide, ARENA is fantastic and I highly recommend it for people interested in learning interpretability!

I'm currently working through the ARENA course now. I completely skipped week 0 because I've done similar content in other courses and university and I'm on section Week 1: Transformer Interpretability now. I'm studying part time so I'm hoping to get through most of the content in a few months.

Reply
The best approaches for mitigating "the intelligence curse" (or gradual disempowerment); my quick guesses at the best object-level interventions
Stephen McAleese1mo41

Some of my thoughts on avoiding the intelligence curse or gradual disempowerment and ensure that humans stay relevant:

  • One solution to ensure that the gap between human and AI intelligence does not grow too large:
    • I think it's often easier to verify solutions than generate them which allows less intelligent agents to supervise more intelligent agents. For example, writing a complex computer program might take 10 hours but checking that the code generally takes ~1 hour and running the program and seeing if it behaves as expected only takes a few minutes. This goal could be achieved by limiting the intelligence of AIs or enhancing human cognitive ability somehow.
  • Devise ways for giving humans a privileged status:
    • AI agents and their outputs will soon vastly outnumber those of humans. Additionally it's becoming impossible to distinguish between the outputs of AIs and humans.
    • One solution to this problem is to make humans more identifiable by watermarking AI outputs (note that watermarks are widely used for paper money) or developing strong proof of human identity (e.g. the blue Twitter mark, iPhone face ID, fingerprint login). This approach is similar to authentication which is a well-known security problem.
    • A short-term solution to differentiating between humans and AIs is to conduct activities in the physical world (although this won't work once sufficiently advanced humanoid robots are developed). For example, voting, exams, and interviews can be carried out in the real world to ensure that participants are human.
    • Once you have solved the problem of differentiating between AI and human outputs, you could upweight the value of human outputs (e.g. writing, art).
Reply
Reward button alignment
Stephen McAleese1moΩ120

After spending some time chatting with Gemini I've learned that a standard model-based RL AGI would probably just be a reward maximizer by default rather than learning complex stable values:

The "goal-content integrity" argument (that an AI might choose not to wirehead to protect its learned task-specific values) requires the AI to be more than just a standard model-based RL agent. It would need:

  1. A model of its own values and how they can change.
  2. A meta-preference for keeping its current values stable, even if changing them could lead to more "reward" as defined by its immediate reward signal.

The values of humans seem to go beyond maximizing reward and include things like preserving personal identity, self-esteem and maintaining a connection between effort and reward which makes the reward button less appealing than it would be to a standard model-based RL AGI.

Reply
Reward button alignment
Stephen McAleese1moΩ120

Thanks for the clarifying comment. I agree with block-quote 8 from your post:

Also, in my proposed setup, the human feedback is “behind the scenes”, without any sensory or other indication of what the primary reward will be before it arrives, like I said above. The AGI presses “send” on its email, then we (with some probability) pause the AGI until we’ve read over the email and assigned a score, and then unpause the AGI with that reward going directly to its virtual brain, such that the reward will feel directly associated with the act of sending the email, from the AGI’s perspective. That way, there isn’t an obvious problematic…target of credit assignment, akin to the [salient reward button]. The AGI will not see a person on video making a motion to press a reward button before the reward arrives, nor will the AGI see a person reacting with a disapproving facial expression before the punishment arrives, nor anything else like that. Sending a good email will just feel satisfying to the AGI, like swallowing food when you’re hungry feels satisfying to us humans.

I think what you're saying is that we want the AI's reward function to be more like the reward circuitry humans have, which is inaccessible and difficult to hack, and less like money which can easily be stolen.

Though I'm not sure why you still don't think this is a good plan. Yes, eventually the AI might discover the reward button but I think TurnTrout's argument is that the AI would have learned stable values around whatever was rewarded while the reward button was hidden (e.g. completing the task) and it wouldn't want to change its values for the sake of goal-content integrity:

We train agents which intelligently optimize for e.g. putting trash away, and this reinforces the trash-putting-away computations, which activate in a broad range of situations so as to steer agents into a future where trash has been put away. An intelligent agent will model the true fact that, if the agent reinforces itself into caring about cognition-updating, then it will no longer navigate to futures where trash is put away. Therefore, it decides to not hit the reward button. 

Though maybe the AI would just prefer the button when it finds it because it yields higher reward.

For example, if you punish cheating on tests, students might learn the value "cheating is wrong" and never cheat again or form a habit of not doing it. Or they might temporarily not do it until there is an opportunity to do it without negative consequences (e.g. the teacher leaves the classroom).

I also agree that "intrinsic" and "instrumental" motivation are more useful categories than "intrinsic" and "extrinsic" for the reasons you described in your comment.

Reply
Load More
Road To AI Safety Excellence
3y
(+3/-2)
16How Can Average People Contribute to AI Safety?
4mo
4
193Shallow review of technical AI safety, 2024
Ω
6mo
Ω
35
23Geoffrey Hinton on the Past, Present, and Future of AI
9mo
5
34Could We Automate AI Alignment Research?
Ω
2y
Ω
10
73An Overview of the AI Safety Funding Situation
2y
10
26Retrospective on ‘GPT-4 Predictions’ After the Release of GPT-4
2y
6
110GPT-4 Predictions
2y
27
3Stephen McAleese's Shortform
2y
13
8AGI as a Black Swan Event
3y
8
47Estimating the Current and Future Number of AI Safety Researchers
3y
14
Load More