In contrast, working on the illegible problems (including by trying to make them more legible) does not have this issue and therefore has a much higher expected value (all else being equal, such as tractability). Note that according to this logic, success in making an illegible problem highly legible is almost as good as solving it!
Obviously if you make an illegible alignment problem is legible, it is at grave risk of being solved - I'm confused as to why you think this is a good thing. Any alignment advance will have capability implications, and so it follows that any alignment advance is bad and to be avoided.
(I.e. I think you're being too paranoid when you worry about solving legible safety problems being net negative)
Wow we have a lot of the same thinking!
I've also felt like people who think we're doomed are basically spending a lot of their effort on sabotaging one of our best bets in the case that we are not doomed, with no clear path to victory in the case where they are correct (how would Anthropic slowing down lead to a global stop?)
And yeah I'm also concerned about competition between DeepMind/Anthropic/SSI/OpenAI - in theory they should all be aligned with each other but as far as I can see they aren't acting like it.
As an aside, I think the extreme pro-slowdown view is something of a vocal minority. I met some Pause AI organizers IRL and brought up the points I brought in my original comment, expecting pushback, but they agreed, saying they were focused on neutrally enforced slowdowns e.g. government action.
My point was that even though we already have an extremely reliable recipe for getting an LLM to understand grammar and syntax, we are not anywhere near a theoretical guarantee for that. The ask for a theoretical guarantee seems impossible to me, even on much easier things that we already know modern AI can do.
When someone asks for an alignment guarantee I'd like them to demonstrate what they mean by showing a guarantee for some simpler thing - like a syntax guarantee for LLMs. I'm not familiar with SLT but I'll believe it when I see it.
Just a note here that I'm appreciating our conversation :) We clearly have very different views right now on what is strategically needed but digging your considered and considerate responses.
Thank you! Same here :)
How do you account for the problem here that Nvidia's and downstream suppliers' investment in GPU hardware innovation and production capacity also went up as a result of the post-ChatGPT race (to the bottom) between tech companies on developing and releasing their LLM versions?
I frankly don't know how to model this somewhat soundly. It's damn complex.
I think it's definitely true that AI-specific compute is further along than it would be if there hadn't been the LLM boom happening. I... (read 470 more words →)
They made a point at the time of expressing concern about AI risk. But what was the difference they made here?
I think you're right that releasing GPT-3 clearly accelerated timelines with no direct safety benefit, although I think there are indirect safety benefits of AI-risk-aware companies leading the frontier.
You could credibly accuse me of shifting the goalposts here, but in GPT-3 and GPT-4's case I think the sooner they came out the better. Part of the reason the counterfactual world where OpenAI/Anthropic/DeepMind had never been founded and LLMs had never been scaled up seems so bad to me is that not only do none of the leading AI companies care about AI... (read more)
I actually doubt there are other general learning techniques out there in math space at all, because I think we're already just doing "approximation of bayesian updating on circuits"
Interesting perspective! I think I agree with this in practice although not in theory (I imagine there are some other ways to make it work, I just think they're very impractical compared to deep learning).
I don't think I can make reliably true claims about anthropic's effects with the amount of information I have, but their effects seem suspiciously business-success-seeking to me, in a way that seems like it isn't prepared to overcome the financial incentives I think are what mostly kill us anyway.
Part of... (read more)
I am in the camp that thinks that it is very good for people concerned about AI risk to be working at the frontier of development. I think it's good to criticize frontier labs who care and pressure them but I really wish it wasn't made with the unhelpful and untrue assertion that it would be better if Anthropic hadn't been founded or supported.
The problem, as I argued in this post, is that people way overvalue accelerating timelines and seem willing to make tremendous sacrifices just to slow things down a small amount. If you advocate that people concerned about AI risk avoid working on AI capabilities, the first order effect of... (read more)
By "it will look like normal deep learning work" I don't mean it will be exactly the same as mainstream capabilities work - e.g. RLHF was both "normal deep learning work" and also notably different from all other RL at the time. Same goes for constitutional AI.
What seems promising to me is paying close attention to how we're training the models and how they behave, thinking about their psychology and how the training influences that psychology, reasoning about how that will change in the next generation.
... (read more)It seems odd and unlikely to me that the same kind of work (normal deep learning) that looks like it causes a series of major problems (power-seeking,
> In order to do useful superalignment research, I suspect you sometimes need to warn about or at least openly discuss the serious threats that are posed by increasingly advanced AI, but the business model of frontier labs depends on pretending that none of those threats are actually serious.
I think this is overly cynical. Demis Hassabis, Sam Altman, and Dario Amodei all signed the statement on AI risk:
"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
They don't talk about it all the time but if someone wants to discuss the serious threats internally, there is plenty of external precedent... (read more)
The alignment community is ostensibly a group of people concerned about AI risk. Lately, it would be more accurate to describe it as a group of people concerned about AI timelines.
AI timelines have some relation to AI risk. Slower timelines mean that people have more time to figure out how to align future intelligent AIs, potentially lowering the risk. But increasingly, slowing down AI progress is becoming an end in itself, taking precedence over ensuring that AI goes well. When the two objectives come into conflict, timelines are winning.
Pretty much anything can be spun as speeding up timelines, either directly or indirectly. Because of this, the alignment community is becoming paralyzed, afraid... (read 2403 more words →)
AI risk is not common knowledge. There are many people who do not believe there's any risk. I really wish people who make arguments of the following form:
would acknowledge this fact. It is simply not true that everyone in frontier labs thinks this way. You can ask them! They'll tell you!
It would be nice if we were just in a prisoner's dilemma type co-ordination problem. But when someone is publicly saying "hitting DEFECT has no downsides whatsoever, I plan on doing that as much as possible" you need to take this into consideration.