LESSWRONG
LW

Joe Carlsmith
5174Ω705108850
Message
Dialogue
Subscribe

Senior advisor at Open Philanthropy. Doctorate in philosophy from the University of Oxford. Opinions my own.

Sequences

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
No wikitag contributions to display.
How might we solve the alignment problem?
Otherness and control in the age of AGI
Scheming AIs: Will AIs fake alignment during training in order to get power?
Can we safely automate alignment research?
Joe Carlsmith2moΩ340

I'm a bit confused about your overall picture here. Sounds like you're thinking something like: 

"almost everything in the world is evaluable via waiting for it to fail and then noticing this. Alignment and bridge-building aren't like this, but most other things are... Also, the way we're going to automate long-horizon tasks is via giving AIs long-term goals. In particular: we'll give them goal 'get long-term human approval/reward', which will lead to good-looking stuff until the AIs take over in order to get more reward. This will work for tons of stuff but not for alignment, because you can't give negative reward for the alignment failure we ultimately care about, which is the AIs taking over." 

Is that roughly right? 

Reply
Can we safely automate alignment research?
Joe Carlsmith2moΩ680

I think it's a fair point that if it turns out that current ML methods are broadly inadequate for automating basically any sophisticated cognitive work (including capabilities research, biology research, etc -- though I'm not clear on your take on whether capabilities research counts as "science" in the sense you have in mind), it may be that whatever new paradigm ends up successful messes with various implicit and explicit assumptions in analyses like the one in the essay. 

That said, I think if we're ignorant about what paradigm will succeed re: automating sophisticated cognitive work and we don't have any story about why alignment research would be harder, it seems like the baseline expectation (modulo scheming) would be that automating alignment is comparably hard (in expectation) to automating these other domains. (I do think, though, that we have reason to expect alignment to be harder even conditional on needing other paradigms, because I think it's reasonable to expect some of the evaluation challenges I discuss in the post to generalize to other regimes.)

Reply
Can we safely automate alignment research?
Joe Carlsmith2moΩ464

I'm happy to say that easy-to-verify vs. hard-to-verify is what ultimately matters, but I think it's important to be clear what about makes something easier vs. harder to verify, so that we can be clear about why alignment might or might not be harder than other domains. And imo empirical feedback loops and formal methods are amongst the most important factors there.

Reply
Can we safely automate alignment research?
Joe Carlsmith2moΩ340

If we assume that the AI isn't scheming to actively withhold empirically/formally verifiable insights from us (I do think this would make life a lot harder), then it seems to me like this is reasonably similar to other domains in which we need to figure out how to elicit as-good-as-human-level suggestions from AIs that we can then evaluate well. E.g., it's not clear to me why this would be all that different from "suggest a new transformer-like architecture that we can then verify improves training efficiency a lot on some metric." 

Or put another way: at least in the context of non-schemers, the thing I'm looking for isn't just "here's a way things could be hard." I'm specifically looking for ways things will be harder than in the context of capabilities (or, to a lesser extent, in other scientific domains where I expect a lot of economic incentives to figure out how to automate top-human-level work). And in that context, generic pessimism about e.g. heavy RL doesn't seem like it's enough.

Reply
Can we safely automate alignment research?
Joe Carlsmith2moΩ340

Sure, maybe there's a band of capability where you can take over but you can't do top-human-level alignment research (and where your takeover plan doesn't involve further capabilities development that requires alignment). It's not the central case I'm focused on, though. 

Also, if there’s an alignment tax (or control tax), then that impacts the comparison, since the AIs doing alignment research are paying that tax whereas the AIs attempting takeover are not.

Is the thought here that the AIs trying to takeover aren't improving their capabilities in a way that requires paying an alignment tax? E.g. if the tax refers to a comparison between (a) rushing forward on capabilities in a way that screws you on alignment vs. (b) pushing forward on capabilities in a way that preserves alignment, AIs that are fooming will want to do (b) as well (though they may have an easier time of it for other reasons). But if it refers to e.g. "humans will place handicaps on AIs that they need to ensure are aligned, including AIs they're trying to use for alignment research, whereas rogue AIs that have also freed themselves human control will also be able to get rid of these handicaps," then yes, that's an advantage the rogue AIs will have (though note that they'll still need to self-filtrate etc). 

Reply
Can we safely automate alignment research?
Joe Carlsmith2moΩ340

Re: examples of why superintelligences create distinctive challenges: superintelligences seem more likely to be schemers, more likely to be able to systematically and successfully mess with the evidence provided by behavioral tests and transparency tools, harder to exert option-control over, better able to identify and pursue strategies humans hadn't thought of, harder to supervise using human labor, etc. 

If you're worried about shell games, it's OK to round off "alignment MVP" to hand-off-ready AI, and to assume that the AIs in question need to be able to pursue make and pursue coherent long-term plans.[1] I don't think the analysis in the essay alters that much (for example, I think very little rests on the idea that you can get by with myopic AIs), and better to err on the side of conservatism. 

I wanted to set aside "hand-off" here because in principle, you don't actually need to hand-off until humans stop being able to meaningfully contribute to the safety/quality of the automated alignment work, which doesn't necessarily need to start around the time we have AIs capable of top-human-level alignment work (e.g., human evaluation of the research, or involvement in other aspects of control -- e.g., providing certain kinds of expensive, trusted supervision -- could persist after that). And when exactly you hand-off depends on a bunch of more detailed, practical trade-offs. 

As I said in the post, one way that humans still being involved might not bottleneck the process is if they're only reviewing the work to figure out whether there's a problem they need to actively intervene on. 

even if human labor is still playing a role in ensuring safety, it doesn’t necessarily need to directly bottleneck the research process – or at least, not if things are going well. For example: in principle, you could allow a fully-automated alignment research process to proceed forward, with humans evaluating the work as it gets produced, but only actively intervening if they identify problems.

And I think you can still likely radically speed up and scale up your alignment research even e.g. you still care about humans reviewing and understanding the work in question. 

  1. ^

    Though for what it's worth I don't think that the task of assessing "is this a good long-term plan for achieving X" needs itself to involve long-term optimization for X. For example: you could do that task over five minutes in exchange for a piece of candy. 

Reply
Can we safely automate alignment research?
Joe Carlsmith2moΩ340

Fair point, and plausible that I'm too much taking for granted a certain subset of development pathways. That is: I'm focused in the essay on threat models that proceed via the automation of capabilities R&D, but it's possible that this isn't necessary.

Reply
Can we safely automate alignment research?
Joe Carlsmith2mo20

By "never build superintelligence" I was assuming we were talking about superintelligent AI, so if the humans in question never build superintelligent AI I'd count this path under that bucket. But as I discussed in my first post in the series, you can indeed get access to the benefits of superintelligence without building superintelligent AI in particular. 

Reply
Can we safely automate alignment research?
Joe Carlsmith2moΩ596

I suspect the author might already agree with all this (the existence of this logical risk, the social dynamics, the conclusion about norms/laws being needed to reduce AI risk beyond some threshold)...

Yes I think I basically agree. That is, I think it's very possible that capabilities research is inherently easier to automate than alignment research; I am very worried about the least cautious actors pushing forward prematurely; as I tried to emphasize in the post, I think capability restraint is extremely important (and: important even if we can successfully automate alignment research); and I think that norms/laws are likely to play an important role there. 

Reply
Can we safely automate alignment research?
Joe Carlsmith2moΩ680

Note that conceptual research, as I'm understanding it, isn't defined by the cognitive skills involved in the research -- i.e., by whether the researchers need to have "conceptual thoughts" like "wait, is this measuring what I think it's measuring?". I agree that normal science involves a ton of conceptual thinking (and many "number-go-up" tasks do too). Rather, conceptual research as I'm understanding it is defined by the tools available for evaluating the research in question.[1] In particular, as I'm understanding it, cases where neither available empirical tests nor formal methods help much.

Thus, in your neurotransmitter example, it does indeed take some kind of "conceptual thinking" to come up with the thought "maybe it actually takes longer for neurotransmitters to get re-absorbed than it takes for them to clear from the cleft." But if some AI presented us with this claim, the question is whether we could evaluate it via some kind of empirical test, which it sounds like we plausibly could. Of course, we do still need to interpret the results of these tests -- e.g., to understand enough about what we're actually trying to measure to notice that e.g. one measurement is getting at it better than another. But we've got rich empirical feedback loops to dialogue with. 

So if we interpret "conceptual work" as conceptual thinking, I do agree that "there will be market pressure to make AI good at conceptual work, because that's a necessary component of normal science." And this is closely related to the comforts I discuss in section 6.1. That is: a lot of alignment research seems pretty comparable to me to the sort of science at stake in e.g. biology, physics, computer science, etc, where I think human evaluation has a decent track record (or at least, a better track record than philosophy/futurism), and where I expect a decent amount of market pressure to resolve evaluation difficulties adequately. So (modulo scheming AIs differentially messing with us in some domains vs. others), at least by the time we're successfully automating these other forms of science, I think we should be reasonably optimistic about automating that kind of alignment research as well. But this still leaves the type of alignment research that looks more centrally like philosophy/futurism, where I think evaluation is additionally challenging, and where the human track record looks worse.

  1. ^

    Thus, from section 6.2.3: "Conceptual research, as I’m understanding it, is defined by the methods available for evaluating it, rather than the cognitive skills involved in producing it. For example: Einstein on relativity was clearly a giant conceptual breakthrough. But because it was evaluable via a combination of empirical predictions and formal methods, it wouldn’t count as 'conceptual research' in my sense."

Reply
Load More
24Video and transcript of talk on AI welfare
2mo
1
78The stakes of AI moral status
2mo
62
21Video and transcript of talk on automating alignment research
Ω
2mo
Ω
0
54Can we safely automate alignment research?
Ω
2mo
Ω
29
78AI for AI safety
Ω
3mo
Ω
13
42Paths and waystations in AI safety
Ω
4mo
Ω
1
20When should we worry about AI power-seeking?
Ω
5mo
Ω
0
31What is it to solve the alignment problem?
5mo
6
63How do we solve the alignment problem?
5mo
9
110Fake thinking and real thinking
5mo
14
Load More