This is a special post for quick takes by HunterJay. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Write for the specific Claude building the specific thing.
I used to think that for publications to influence the labs, an employee would need to read and implement the ideas.
But now, I think that the one reading and implementing the work is Claude, and Claude can read a lot more stuff. If Claude is asked to build a new RL environment, he could read every relevant paper before starting the project, whereas a human engineer might only read one or two, if any.
This means that if you are writing and publishing research, if it's net positive work at all, it's way more likely to be read and considered at implementation time than it was in the past (when it might have been ignored completely, or just vaguely remembered).
This updates me:
Four upvotes on the Alignment Forum could be enough to have an idea be useful.
When I ask Claude to help me with AI safety research, I usually already have a pretty good idea of what I want it to implement. It doesn't usually look up papers unless I ask it to, and I probably wouldn't want it to bring in ideas from random papers it just read that I don't understand, since they might mess up my results.
If I were having Claude implement RL environments to train a frontier model, it may be somewhat different: it could make sense to bring in a whole kitchen sink of ideas for improving the model's alignment. But I'd still be pretty wary about it. I'd be okay with Claude suggesting a new technique, but I'd at least want to read a few paragraphs explaining what it is and how it works. I'd also be much more likely to consider a technique if it had gained popularity among humans: in the hypothetical world where e.g. inoculation prompting never got much attention, I wouldn't want Claude to proactively add it.
In the future, AI developers will probably trust their AIs to independently research random ideas and get more evidence about whether they should be implemented widely. The AIs can comb through many 4-upvote LW posts and comments, invest effort in the ones they think are promising, and only present the most important results to the humans. See my shortform about scraping LessWrong for AI safety ideas. Once humans agree that certain ideas are good, they can be implemented systematically across a whole training run.
So in conclusion, I agree with you in spirit, but I think that ideas will usually influence decisions made by an organization or the whole AI safety community, rather than influencing individual Claudes spun up by individual users.
Aye, I think we're broadly in agreement, and I don't claim that this is a perfect system or guaranteed to get useful ideas implemented in labs, just that it's easier than I previously thought.
That said, I think an important idea here is that it's likely to happen more as AIs do more work (which you gesture at too). To riff a bit, if we map increasing autonomy onto your LessWrong scraping idea, I could imagine it going from "Find and read the research and show me your top picks" to "Find and read the research, replicate your top picks and assess them for our specific project, and then show me your results from that" to "On a weekly cron job, find and read any research relevant to our projects, run them through the assessment pipeline, analyse the results yourself, and implement anything you think is good automatically".
The last stage is a big jump, but will happen at some point as we approach RSI. I expect it to happen soonest in areas that are verifiable, especially where there is a big cost to not making a change quickly (e.g. searching for methods to make sandboxes more secure and allowing Claude to automatically patch them), or where there's little cost to getting it wrong (e.g. automated research agents taking ideas and implementing them in experimental small scale runs in different combinations / conditions / scale to the original work).
That last one admittedly still goes through the human bottleneck, but at a later stage, and I think that touches on your point about ideas influencing the organization / community level. If a piece of work was good but not great (e.g. had some interesting ideas, but wasn't perfectly implemented, or was badly presented, or not done in realistic conditions), it is now more likely to be elevated by a research agent in a lab investigating the same thing.
My original post spoke about an instance of Claude trying to implement a specific project, finding the research, and implementing it in its final run, which is somewhat different to the research agent example or the LessWrong scraping example, but I think they all point in the same direction of outside, publicly available work having more value because of the ability of AIs to quickly read, assess, replicate, and implement it at a much larger scale than traditional human engineering.
I know there's a lot of AI news digests, but I built yet another one in my spare time to cover the things that I wanted to hear about.
It's a public thing, so if you want another daily AI digest, this one sends out every morning just before 9am AEST, scrapes all of the good substacks, twitter accounts, and orgs, as well as LessWrong, Alignment Forum, and Manifold Markets, and figures out the most relevant news from a trends + policy + AI safety perspective, then links a top story + a couple of others.
It's 95% AI generated, but I do get an early edition and have set it up so the AI running it receives my emailed feedback and has a go at acting on it before the main one goes out each morning.
Early days, so please forgive the rough edges and somewhat embarrassing AI generated copy. Feedback / missed sources very welcome.
Just wanted to bump this and say that the daily digest continues to improve!
Regarding AGI race dynamics -- I wonder if there's an intuition pump for 'time vs competitor' preference?
For example, to me, based on my current knowledge, I think Anthropic reaching RSI before the next best company (Deepmind, maybe?) is worth about two years of time. (I.e. I estimate equal safety-relevant outcomes from Claude hitting RSI in 2027 as from Gemini hitting RSI in 2029).
That's a super weird framework, and I just made up that two years number, but I think maybe helps me reason through preferences.
The neat thing about the framework is that it's p(doom) agnostic. It's about relative performance between AGI projects and expectations for how much safety work will reduce it in the near future, absolute numbers not needed.
It also lets you give clear, recordable, updatable beliefs. So, spitballing:
Anthropic -- Leader
Deepmind -- +2 years for equal safety
OpenAI -- +2.5 years
SSI -- +2.5 years
Deepseek -- +3 years
Zai -- +4 years
Xai -- +5 years
Alibaba -- +5 years
Meta -- +6 years
Again I want to stress these are vibes, not a considered opinion. I expect to change my mind quickly once challenged with evidence my guesses are wrong.
Noticing this again, I now have much more compressed times between the labs. I wonder if that's what all the disagreement votes were?
Anthropic -- Leader
OpenAI -- +4 months for equal safety
Deepmind -- +4 months
SSI -- +4 months
Deepseek -- +6 months
Zai -- +6 months
Xai -- +12 months
Alibaba -- +12 months
Meta -- +12 months
We might be able to model some idealised AI transition as people bidding on intelligence. AIs carry out increasingly open ended and underspecified work trying to do what's best for the specific human, given some amount of token usage the human bids for.
I think this continuous model is cleaner and more accurate than a binary 'do you have access to this model or not' (although in reality it will not be a clean auction system because people have other preferences about how they want prices to behave and can impose social costs to encourage sellers to set prices more arbitrarily.)
The price mechanism here would surface high value uses effectively during this transition. People saying 'This is my problem and I will pay to have it solved' gives information that anything short of transformative AI won't have.
You could model the AI company as having a bid price of its own, where below this number it would prefer to receive the cash than over additional AI usage. You would then expect the labs to still use the AI a lot, but if there is diminishing marginal return, and need for revenue, there must be a cross-over point.
They also have an incentive to provide AI services broadly, because they benefit indirectly from the rest of the economy running more efficiently. Neither they, nor the AI, can see what the highest value uses of the AI are overall, so by handing this off to a price mechanism, they get access to the value generated via the improvements in the broader economy (their power costs go down, their food becomes cheaper, their suppliers and their supplier's supplier's run more efficiently, etc, etc).
In reality, there are forces against the labs providing AI services like this too, such as caring about the relative gap in capabilities between themselves and their competitors or bad actors.