This is a special post for quick takes by Garrett Baker. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
A list of some contrarian takes I have:
-
People are currently predictably too worried about misuse risks
-
What people really mean by "open source" vs "closed source" labs is actually "responsible" vs "irresponsible" labs, which is not affected by regulations targeting open source model deployment.
-
Neuroscience as an outer alignment[1] strategy is embarrassingly underrated.
-
Better information security at labs is not clearly a good thing, and if we're worried about great power conflict, probably a bad thing.
-
Much research on deception (Anthropic's recent work, trojans, jailbreaks, etc) is not targeting "real" instrumentally convergent deception reasoning, but learned heuristics. Not bad in itself, but IMO this places heavy asterisks on the results they can get.
-
ML robustness research (like FAR Labs' Go stuff) does not help with alignment, and helps moderately for capabilities.
-
The field of ML is a bad field to take epistemic lessons from. Note I don't talk about the results from ML.
-
ARC's MAD seems doomed to fail.
-
People in alignment put too much faith in the general factor g. It exists, and is powerful, but is not all-consuming or all-predicting. People are often very smart, but lack social skills, or agency, or strategic awareness, etc. And vice-versa. They can also be very smart in a particular area, but dumb in other areas. This is relevant for hiring & deference, but less for object-level alignment.
-
People are too swayed by rhetoric in general, and alignment, rationality, & EA too, but in different ways, and admittedly to a lesser extent than the general population. People should fight against this more than they seem to (which is not really at all, except for the most overt of cases). For example, I see nobody saying they don't change their minds on account of Scott Alexander because he's too powerful a rhetorician. Ditto for Eliezer, since he is also a great rhetorician. In contrast, Robin Hanson is a famously terrible rhetorician, so people should listen to him more.
-
There is a technocratic tendency in strategic thinking around alignment (I think partially inherited from OpenPhil, but also smart people are likely just more likely to think this way) which biases people towards more simple & brittle top-down models without recognizing how brittle those models are.
A non-exact term ↩︎
20
3
2
2
1
1
1Showing 3 of 10 replies (Click to show all)
1
70
Ah yes, another contrarian opinion I have:
- Big AGI corporations, like Anthropic, should by-default make much of their AGI alignment research private, and not share it with competing labs. Why? So it can remain a private good, and in the off-chance such research can be expected to be profitable, those labs & investors can be rewarded for that research.
14
I talked about this with Garrett; I'm unpacking the above comment and summarizing our discussions here.
* Sleeper Agents is very much in the "learned heuristics" category, given that we are explicitly training the behavior in the model. Corollary: the underlying mechanisms for sleeper-agents-behavior and instrumentally convergent deception are presumably wildly different(!), so it's not obvious how valid inference one can make from the results
* Consider framing Sleeper Agents as training a trojan instead of as an example of deception. See also Dan Hendycks' comment.
* Much of existing work on deception suffers from "you told the model to be deceptive, and now it deceives, of course that happens"
* (Garrett thought that the Uncovering Deceptive Tendencies paper has much less of this issue, so yay)
* There is very little work on actual instrumentally convergent deception(!) - a lot of work falls into the "learned heuristics" category or the failure in the previous bullet point
* People are prone to conflate between "shallow, trained deception" (e.g. sycophancy: "you rewarded the model for leaning into the user's political biases, of course it will start leaning into users' political biases") and instrumentally convergent deception
* (For more on this, see also my writings here and here. My writings fail to discuss the most shallow versions of deception, however.)
Also, we talked a bit about
[...]
and I interpreted Garrett saying that people often consider too few and shallow hypotheses for their observations, and are loose with verifying whether their hypotheses are correct.
Example 1: I think the Uncovering Deceptive Tendencies paper has some of this failure mode. E.g. in experiment A we considered four hypotheses to explain our observations, and these hypotheses are quite shallow/broad (e.g. "deception" includes both very shallow deception and instrumentally convergent deception).
Example 2: People generally seem to have an opinion of "cha