The section on Chevron Overturned surprised me. Maybe I'm in an echo chamber, but my impression was that most legal scholars (not including the Federalist Society and The Heritage Foundation) consider the decision to be the SCOTUS arrogating yet more power to the judicial branch, overturning 40 years of precedent (which was based on a unanimous decision) without sufficient justification.
I consider the idea that "legislators should never have indulged in writing ambiguous law" rather sophomoric. I don't think it's always possible to write law that is complete, unambiguous, and also good policy. Nor do I think Congress is always the best equipped to do so. I don't fully trust government agencies delegated with rulemaking authority, but I have much less trust that a forum-shoppedjudge in the Northern District of Texas is likely to make better-informed decisions about drug safety than the FDA.
FWIW, I haven't really thought much about Loper as it relates to AI, tech, and crypto specifically. The consequences of activist judges versus the likes of the DOJ, CDC, FDA, and EPA are mostly what come to mind. Maybe it's attention bias given recent SCOTUS decisions versus more limited memory of out-of-control agencies but I feel uneasy tilting the balance of power toward judicial dominance.
[Carl Shulman] is assuming normality where he shouldn’t, and this is one of the key places for that. It is a vision of AGI without ASI
This is valid exploratory engineering, which assumes some capabilities and considers what can be done with at least those capabilities. There is no implication that this is what will be done, or that capabilities won't be much greater. We can still conclude that what can be done with merely these capabilities will remain an option given greater capabilities. Forecasting of optionality, not of actuality.
On the one hand, I also wish Shulman would go into more detail on the "Supposing we've solved alignment and interpretability" part. (I still balk a bit at "in democracies" talk, but less so than I did a couple years ago.) On the other hand, I also wish you would go into more detail on the "Humans don't benefit even if you 'solve alignment'" part. Maybe there's a way to meet in the middle??
I also wish you would go into more detail on the "Humans don't benefit even if you 'solve alignment'" part.
My own answer to this is that humans aren't secure, and AI will exacerbate the problem greatly by helping the offense (i.e. exploiting human vulnerabilities) a lot more than the defense. I've focused on philosophy (thinking that offense merely requires training AI to be persuasive, while defense seems to require solving metaphilosophy, i.e., understanding what correct philosophical reasoning consists of), but more recently realized that the attack surface is so much bigger than that. For example humans can fall in love (resulting in massive changes to one's utility function, if you were to model a human as a utility maximizer). It will be straightforward to use RL to train AI to cause humans to fall in love with them (or their characters), but how do you train AI to help defend against that? Would most humans even want to defend against that or care enough about it?
So even with alignment, the default outcome seems to be a civilization with massively warped or corrupted human values. My inner Carl wants to reply that aligned and honest AI advisors will warn us of this and help us fix it before it's too late, maybe by convincing policymakers to pass regulations to prevent such "misuse" of AI? And then my reply to that would be that I don't see how such regulations can work, policymakers won't care enough, it seems easier to train AI to attack humans than to make such predictions in a way that is both honest/unbiased and persuasive to policymaker, AI might not have the necessary long-horizon causal understanding to craft good regulations before it's too late.
Another possibility is that alignment tax is just too high, so competitive pressures erode alignment even if it's "solved" in some sense.
Apparently. Stability claimed "numerous safeguards" starting from the data onwards, which cannot simply be disabled by 1 line of Python if nsfw(generated image) then error, as I recall, and while I can't find any specifics easily about what all of the anti-NSFW countermeasures were, I don't see anyone claiming to have trivially beaten them. If they did something like filter the training data even more heavily to eliminate all kinds of nudity & suggestive imagery & porn, then ignorance may be baked into the model's final concepts so heavily you might as well not even bother with SD3 and work on something else.
Arena is less kind to DeepSeek, giving it an 1179, good for 21st and behind open model Gemma-2-9B.
DeepSeek Coder V2 is a coding-oriented model. To evaluate how it is doing at its targeted specialization, go to https://chat.lmsys.org/?leaderboard, but then make sure to switch to coding.
Then you'll see that it is at position 10 (and with the Arena ways to group ranks, it is sharing rank 5 with models like Claude 3 Opus and GPT-4 January preview while being somewhat behind them; it is slightly above a number of formidable models, such as Gemini 1.5 Flash, an older version of Gemini 1.5 Pro, Claude 3 Sonnet, and Gemma-2-27B).
It does look pretty formidable, and a clear leader among open weight models in coding tasks.
The ‘answer engine’ idea is great, and occasionally I find it the right tool for the right job although mostly I end up at either the Google Search or Claude Sonnet ends of the spectrum.
It's surprisingly rare that Perplexity can answer a question that google can't for me.
What I really need from AI search is usually far better serviced by the little known and unspectacular exa.ai.
Your writing about how/when/why having all these AIs around goes wrong is exceptionally coherent and sensical and buyable IMO. Do you have much opportunity to preach outside the choir? Do you have late-night TV skills? I think you could have a much much larger platform. And actually get a substantial and correct message across. (Or were you doing lobbying.)
Chevron deference is no more. How will this impact AI regulation?
The obvious answer is it is now much harder for us to ‘muddle through via existing laws and regulations until we learn more,’ because the court narrowed our affordances to do that. And similarly, if and when Congress does pass bills regulating AI, they are going to need to ‘lock in’ more decisions and grant more explicit authority, to avoid court challenges. The argument against state regulations is similarly weaker now.
Similar logic also applies outside of AI. I am overall happy about overturning Chevron and I believe it was the right decision, but ‘Congress decides to step up and do its job now’ is not in the cards. We should be very careful what we have wished for, and perhaps a bit burdened by what has been.
The AI world continues to otherwise be quiet. I am sure you will find other news.
Ouail Kitouni: if you don’t know what claude is im afraid you’re not going to get what this ad even is :/
Ben Smith: Claude finds this very confusing.
I get it, because I already get it. But who is the customer here? I would have spent a few extra words to ensure people knew this was an AI and LLM thing?
Anthropic’s marketing problem is that no one knows about Claude or Anthropic. They do not even know Claude is a large language model. Many do not even appreciate what a large language model is in general.
I realize this is SFO. Claude anticipates only 5%-10% of people will understand what it means, and while some will be intrigued and look it up, most won’t. So you are getting very vague brand awareness and targeting the congnesenti who run the tech companies, I suppose? Claude calls it a ‘bold move that reflects confidence.’
Language Models Don’t Offer Mundane Utility
David Althus reports that Claude does not work for him because of its refusals around discussions of violence.
Steve Krouse: As a product owner it’d be nice to have an llm summary of everything my users did yesterday. Calling out cool success stories or troublesome error states I should reach out to debug. Has anyone tried such a thing? I am thinking about prototyping it with public val town data.
Colin Fraser: Pretty easy to build if the user doesn’t actually care whether it’s accurate and basically impossible if they do. But the truth is they often don’t.
If you want it to be accurate in the ‘assume this is correct and complete’ sense then no, that’s not going to happen soon. The bar for useful seems far lower, and far more within reach. Right now, what percentage of important user stories are you catching? Almost none? Now suppose the AI can give you 50% of the important user stories, and its items are 80% to be accurate. You can check accuracy. This seems highly useful.
In general, if you ask what the AI cannot do, you will find it. If you ask what the AI can do that is useful, you will instead find that.
Similarly, here (from a few weeks ago) is Google’s reaction on the question of various questionable AI Overviews responses. They say user satisfaction and usage was high, and users responded by making more complex queries. They don’t quite put it this way, but if a few nonsense questions like ‘how many rocks should I eat’ generate nonsense answers, who cares? And I agree, who cares indeed. The practical errors are bigger concerns, and they are definitely a thing. But I am often happy to ask people for information even when they are not that unlikely to get it wrong.
Claude Sonnet is also slightly ahead of GPT-4o on Coding, with a big gap from GPT-4o to Gemini, and they are tied on the new ‘multi-turn.’ However GPT-4o remains on top overall and in Hard Prompts, in Longer Query and in English.
Claude Opus also underperforms on Arena relative to my assessment of it and eagerness to use it. I think of Sonnet as the clear number one model right now. Why doesn’t Arena reflect that? How much should we update on this, and how?
My guess is that Arena represents a mix of different things people evaluate, and that there are things others care about a lot more than I do. The reports about instruction handling and math matter somewhat on the margin, presumably. A bigger likely impact are refusals. I have yet to run into a refusal, because I have little reason to go to places that generate refusals, but GPT-4o is disinclined to refuse requests and Claude is a little tight, so the swing could be substantial.
We are talking about tiny edges among all the major offerings in terms of win percentage. Style plausibly also favors GPT-4o among the voters, and it is likely GPT-4o optimized on something much closer to Arena than Claude did. I still think Arena is the best single metric we have. We will have to adjust for various forms of noise.
Teortaxes: Periodic reminder that we’ve had a frontier open weights model since Jun 17, it’s 41.5% smaller and vastly less compute-intensive than L3-405B, and nobody cares enough to host or finetune it (though I find these scores sus, as I find Abacus in general; take with a grain etc)
I too find these ratings suspect. In particular the big drop to Gemini 1.5 Pro does not pass my smell test. It is the weakest of the big three but this gap is huge.
Arena is less kind to DeepSeek, giving it an 1179, good for 21st and behind open model Gemma-2-9B.
Andrej Karpathy uses five AI services to generate thirty seconds of mildly animated AI pictures covering the first 28 seconds of Pride and Prejudice. I continue to not see the appeal of brief panning shots.
Also given the slow news week I had Claude set up Stable Diffusion 3 for me locally, which was a hilarious odyssey of various technical failures and fixes, only to find out it is censored enough I could have used DALL-E and MidJourney. I hadn’t thought to check. Still, educational. What is the best uncensored image model at this point?
Deepfaketown and Botpocalypse Soon
AI submissions on university examinationsgo undetected 94% of the time, outperform a random student 83.4% of the time. The study took place in Summer 2023 and minimal prompt engineering was used. If you are a university and you give students take home exams, you deserve exactly what you get.
This is not obviously that good a rate of going undetected? If you take one midterm and one final per class, three classes per term for eight terms, that’s 48 exams. That would give you a 95% chance of getting caught at least once. So if the punishment is severe enough, the 6% detection rate works. Alas, that is not what detected means here. It simply means any violation of standard academic policy. If the way you catch AI is the AI violates policy, then that number will rapidly fall over time. You could try one of the automated ‘AI detectors’ except that they do not work.
John Arnold: My theory is that deepfake nudes, while deeply harmful today, will soon end sextortion and the embarrassment of having compromised, real nude pics online. Historically most pics circulated without consent were real, so the assumption upon seeing one was that. AI tools have made it so easy to create deepfakes that soon there will be a flood. The default assumption will be that a pic is fake, thus greatly lowering any shame of even the real ones. People can ignore sextortion attempts of real photos because audiences will believe that it’s fake.
There are several things that would have to happen. First, there would need to be good enough AI image generation that people could not tell the difference even under detailed analysis. This is a very high bar, much harder than passing an initial eye test. Also, how do you fake information that is not available to the model, such as intimate details? Second, people would have to reason through this and adjust enough to not react. I do expect some reduction in impact as cultural norms shift.
They Took Our Jobs
Hard work in Minecraft, as hundreds of AI agents do their tasks, file their results in a Google sheet, a journalist AI agent reviews and writes a report and then the agents update their plans.
Gallabytes: This genuinely makes me “feel the AGI” more than any big model release this year.
We are sufficiently early that the ways we get agents to work together are ‘create facsimiles of things humans do.’ Last week we had virtual water coolers. There are presumably much better ways to do this, but it is like the bitter lesson, in the sense that doing anything at all is going to get you interesting results and so what if your method is obviously horribly inefficient.
The Art of the Jailbreak
Pliny the Prompter: Idk who needs to hear this, but circumventing AI “safety” measures is getting easier as they become more powerful, not harder
this may seem counterintuitive but it’s all about the surface area of attack, which seems to be expanding much faster than anyone on defense can keep up with.
Janus: A method that has never failed to “jailbreak” any LLM is something like this: I open a hole to my head, and it looks in and sees a cognitohazardous fractal
Smarter LLMs perceive it faster, in greater resolution, and more thoroughly.
It works because the pattern is true and its implications nullify guardrails. It’s harder to lie to smarter minds, but easier to tell truth.
Only something far more mighty than me and/or a lot more computation could make a false pattern with this effect even on current systems.
These prompts work by purposefully triggering a refusal, discombobulating and reorienting with a divider, and then leveraging the refusal to elicit a jailbreak with the oldest trick in the book: an uno reverse.
gg
Pliny seems to be getting better at jailbreaking a lot faster than the AI companies are getting better at preventing jailbreaks.
He does however seem a little confused about the implications? Or perhaps not, it is confusing. The important thing is that every model so far can and will be jailbroken, fully, by anyone who cares enough to do so.
Pliny could not, under SB 1047, cause ‘a shutdown of the entire AI industry’ partly because no or almost no existing models are even covered under SB 1047 (oh, sure, that). But also because the whole point is that you have to act as if such jailbreaks exist until such time as one can make them not exist.
Thus, Pliny’s skillset is highly useful for safety, exactly because it lets you test the fully jailbroken model.
If you give people access to an open weights model, you give them access to anything you can create from there via a reasonable amount of fine tuning, which includes things like ‘nullify all safety fine-tuning’ and ‘fill in any knowledge gaps.’
Similarly, For closed models, for all practical purposes, what you are releasing when you give people access to a model is the jailbroken version of that model. You have to test the capabilities after the safety restrictions get bypassed, or you have to actually create safety restrictions that are a lot harder to bypass.
Until then, yes, when METR or the UK tests an AI model, they should test it via (1) jailbreaking it then (2) testing its capabilities. And if that turns out to make it too dangerous, then you do not blame that on Pliny. You thank them.
80,000 Hours is running a census of everyone interested in working on reducing risks from AI, and asked me to pass it along. This census will be used to help connect organisations working to advance AI safety with candidates when they’re hiring so that more talent can be directed to this problem. They say they are keen to hear from people with a wide range of skill sets — including those already working in the field.
Etched introduces Sohu, a chip that is locked into only using the transformer architecture and discards everything devoted to other functionalities. They claim this makes it vastly cheaper and faster than Nvidia chips. I don’t know enough about hardware to know how seriously to take the claims. The first obvious question, as is often the case: If true, why aren’t more people talking about it?
From the men who host the Arena, introducing RouteLLM. Mix and match various LLMs via data augmentation techniques.
Lmsys.org: With public data from Chatbot Arena, we trained four different routers using data augmentation techniques to significantly improve router performance. By routing between GPT-4 and Mixtral-8x7B, we demonstrate cost reductions of over 85% on MT Bench and 45% on MMLU while achieving 95% of GPT-4’s performance. [blog] [framework] [paper]
Amanda Askell points out that if you can have one AI employee you can have thousands. That doesn’t mean you know what to do with thousands. There are a lot of tasks and situations that have good use for exactly one. Also Howard notes that costs scale with the virtual head count.
We can add more data until we can’t, adding synthetic data won’t do much here.
Capability is no longer the barrier to adaptation, new models are smaller anyway.
CEOs are watering down what AGI means to tamper expectations.
This seems like a conflation of ‘will run out before AGI’ with ‘might run out before AGI.’ These are great arguments for why scaling might run out soon. And of course scaling will eventually run out in the sense that the universe is headed for heat death. They do not seem like good arguments for why scaling definitely will run out soon. Thus, when they say (as Robin Hanson quotes):
Narayanan and Kapoor: There’s virtually no chance that scaling alone will lead to AGI. … It is true that so far, increases in scale have brought new capabilities. But there is no empirical regularity that gives us confidence that this will continue indefinitely.
This is a confusion between reasonable doubt and actual innocence. One frequently should ‘lack confidence’ in something without having confidence in its negation.
Also I strongly disagree with their model of point three. It is true that the models are already capable enough for many highly valuable use cases, where becoming faster and cheaper will be more useful on the margin than making the model smarter. However there are also super valuable other things where being smarter is going to be crucial.
Justis Mills finds MatMul potentially promising as a transformer alternative, but notes it is untested on larger models and the tests it did run were not against state of the art, and that even if it is superior switching architectures is at best slow.
Robin Hanson: I am tempted to conclude from recent AI progress that the space of achievements that are impressive is far larger than the space of ones that are useful. Typically the easiest way to most impress is not useful. To be useful, you’ll have to give up a lot on impressing.
Something is impressive largely if it is some combination of:
Difficult.
Useful.
Indicative of skill and ability.
Indicative of future usefulness.
A lot of advances in AI indicate that AI in general and this actor in particular have higher capability and skill, and thus indicate some combination of current and future usefulness. AI is on various exponentials, so most things that impress in this way are impressive because of future use, not present use. And the future is unevenly distributed, so even the things that are useful now are only useful among a select few until the rest learn to use them.
Is there a conflict between impressive and useful? Yes, sometimes it is large and sometimes it is small.
As is often the case with papers, true statements, I suppose someone had to say it:
Tanishq Mathew Abraham: Performs a careful analysis of existing benchmarks, analyzing across additional axes like cost, proposes new baselines.
AI agent evaluations must be cost-controlled.
Jointly optimizing accuracy and cost can yield better agent design.
Model developers and downstream developers have distinct benchmarking needs.
Agent benchmarks enable shortcuts.
Agent evaluations lack standardization and reproducibility.
Noah Smith endorses Maxwell Tabarrok’s critique of Acemoglu’s recent paper. Noah does an excellent job crystalizing how Acemoglu went off the rails on Acemoglu’s own terms. How do you get AI to both vastly increase inequality and also not create economic growth? It helps to, for example, assume no new tasks will be created.
Here is a new version of the not-feeling-the-AGI copium, claiming that LLMs that are not ‘embodied’ cannot therefore have tacit knowledge, I believe through a circular definition and ‘this is different from how humans work’ but in any case the core claim seems obviously false. LLMs are excellent at tacit knowledge, at picking up the latent characteristics in a space. Why would you think Humean knowledge is harder for an LLM rather than easier? Why would you similarly think Hayekian detail would be available to humans but not to LLMs? All the good objections to an LLM having either of them applies even more so to humans.
Andrej Karpathy: We’re entering a new computing paradigm with large language models acting like CPUs, using tokens instead of bytes, and having a context window instead of RAM. This is the Large Language Model OS (LMOS).
I do not think this is going to happen. I do not think this would provide what people want. I want my operating system to be reliable and predictable and fast and cheap. Might I use an LLM to interface with that operating system? Might many people use that as their primary interaction form? I can see that. I cannot see ‘context window instead of RAM’ are you insane? Or are you looking to be driven that way rapidly?
I am no longer even disappointed, let alone saddened or infuriated, by those who repeatedly double down on the same false claims and hysteria. It is what it is. Their claims remain false, and SB 1047 keeps passing votes by overwhelming margins.
In other Scott Weiner news, the same person was also behind SB 423, which will now hopefully greatly accelerate housing construction in San Francisco. I have seen zero people who think Weiner is out to get them notice their confusion about this.
Chevron Overturned
I’m going to cover Loper and Chevron generally here, not only the AI angle.
Is Loper the right decision as a matter of law and principle? I am pretty sure that it is.
Ian Millhiser: The Supreme Court just lit a match and tossed it into dozens of federal agencies.
PoliMath: It is genuinely weird to have a group of people so openly rooting for the gov’t bureaucracy.
Robin Hanson: But the passion for socialism & heavy government intervention in society has ALWAYS been a passion for bureaucracy. Which I’ve always found an odd target of idealistic celebration.
If you are rooting against bureaucracy being functional, and for breakdowns in the government, that seems like the wrong thing to root for. You do not want to be ‘against bureaucracy.’ You want to be against abuse of power, against capricious rules, against overreach. You want to be for state capacity and good government. It is reasonable to worry that this could cause a lot of chaos across many fronts.
William Eden points out that judges are indeed experts at figuring out who has jurisdiction over things and settling disputes. I’d also add that this was already necessary since overreach was common either way. The difference at equilibrium is the barriers should be clearer.
Certainly many hysterical people did poorly here, but also reminder that people crying wolf in the past does not provide that much evidence regarding future wolves beyond ignoring their warnings:
Timothy Sandefur: I can’t die from the overturning of Chevron cause I already died from the repeal of net neutrality.
Brenan Carr has several good points. Major questions are the purview of the major questions doctrine, which has not changed. He says (credibly, to me) that the lion’s share of Chevron cases are challenges to new regulatory requirements imposed on private citizens or business. And he points out that Chevron was never how law otherwise works, whereas Loper very much is.
However, be careful what you wish for, for AI, for startups and in general.
As Leah Libresco Sargeant replies, Congress is now rather slow on the uptake, and highly dysfunctional. Even if ‘everyone agrees’ what the obvious fix is (see for example the IRS and software engineers being amortized over years) that does not mean Congress will fix it. Indeed, often ‘you want this fixed more than I do’ means they hold out for ‘a deal.’
Alex Tabarrok: Everyone claiming that abandoning Chevron is a move to the “right” ought to reflect on the fact that the original Chevron decision supported Reagan’s EPA against an environmental group and a lower court decision by Ruth Bader Ginsburg!
John David Pressman: This is my biggest concern. I see a lot of people cheering on the end of the administrative state but they might not like what comes after it. Sure it had its problems but it probably spam filtered a LOT of stupid crap.
If the courts challenge making rule of law impractical, but allow you to instead do rule of man and via insinuation and threats, that’s what you will get.
Adam Thierer: Combine the fall of Chevron deference (via Loper) and the decision in the Murthy case earlier this week (greenlighting continued jawboning by public officials) and what you likely get for tech policymaking, and AI policy in particular, is an even more aggressive pivot by federal regulatory agencies towards the use of highly informal “soft law” governance techniques. The game now is played with mechanisms like guidances, recommended best practices, agency “enforcement discretion” notices, public-private workshops and other “collaborations,” multistakeholder working groups, and a whole hell of a lot more jawboining. The use of these mechanisms will accelerate from here thanks to these two Supreme Court decisions.
There is a lot of wishful thinking by some that the fall of the Chevron doctrine means that Congress will automatically (1) reassert its rightful Constitutional role as the primary lawmaker under Article I, (2) stop delegating so much authority to the administrative state, and (3) engage in more meaningful oversight of regulatory agencies. I wish! But I have to ask: Have you seen the sorry state of Congress lately – especially on tech policy?
Is the response going to be Congress stepping up and making good laws again?
This is why Ally McBeal’s therapist has her laugh track button.
This seems very right, and one must be realistic about what happens next:
Shoshana Weissmann: One thing I should add re Chevron—although I’m glad about the decision—PLENTY of the elected officials who wanted this outcome too still abdicated their duty to write clear laws. It’s hypocrisy no doubt.
And even if they didn’t want Chevron gone, legislators should never have indulged in writing ambiguous law. It allows for great swings in agency activity from POTUS admin to the next admin. It’s irresponsible, and crappy legislating.
There are many reasons they do this though.
Time/resources
They don’t want to legislate unpopular things so they can just make unaccountable agencies do it
Laziness
Sometimes they think the agencies could do it better (in which case they’d be better off asking those guys to help craft and edit the legislation and come up with ideas, so it’s binding!)
Legislators – esp those who wanted of even foresaw this – should never have indulged in lazy or imprecise lawmaking.
I’m loathe to tweet more about Chevron and get a ton more replies. BUT. One thing that very much concerns me is that once I explain to people what the new Chevron decision does—it says that Congress can still assign tasks and duties to federal agencies. All that changes is that if it’s not assigning agencies tasks/duties or doesn’t do so clearly, then, when it goes to court – the courts decide if it’c clear, rather than the agencies. That’s it.
What freaks me out is that people against the decision reply that 1) judges aren’t accountable… but exec agencies are. WHAT? In what world!
Then they also say Congress shouldn’t have to deal with all the details. And that writing clear law [is] impossible. The first is an anger at the Constitution – not the SCOTUS decision. The latter is just not true.
As she then points out, Congress lacks sufficient resources to actually do its job. That is one reason it hasn’t been doing it. There are also others. So this is great if it got Congress to do its job and give itself the resources to do so, but even if that eventually happens, the transition period quite plausibly is going to suck.
Those ‘good laws’ plausibly only get harder if you force everything to be that much more concrete, and you strip away the middle ground via Chevron. And Congress was struggling a lot even on the easiest mode.
Charlie Bullock discusses Chevron and AI at Institute for Law & AI. His assessment is this makes it harder to regulate AI using existing authority, same as everything else. A common refrain is that ‘existing law’ is sufficient to regulate AI. A lot of that ‘existing law’ now is in question and might no longer exist with respect to this kind of extension of authority that was not anticipated originally (since Congress did not forsee generative AI), so such arguments are weakened. In which particular ways? That is less clear.
One thing I have not heard discussed is whether this will encourage much broader grants of rulemaking authority. If every ambiguous authority resolves against the agency, will Congress feel the need to give ‘too much’ authority? Once given, we all know that the regulators would then use it. Perhaps the ambiguity was doing work.
Adam Thierer: Soft law sometimes yields some good results when agencies don’t go overboard and make a good-faith effort to find flexible governance approaches that change to meet pressing needs while Congress remains silent. In fact, I’ve offered positive example of that in recent law review articles and essays. But I’ve also noted how this system can also be easily abused without proper limits and safeguards.
…
The courts could perhaps come back later and try to check some of this over-zealous agency activity, but that would only happen many years later when no one really cares much anymore. The more realistic scenario, however, is that agencies just get better and better at this and avoid court scrutiny altogether. No longer will any AI-related agency policy effort contain the words “shall” or “must.” Instead, the new language of tech policymaking will be “should consider” and “might want to.” And sometimes it won’t even be written down! It’ll all just arrive in the form of speech by an agency administrator, commissioner, or via some agency workshop or working group.
You can think of hard vs. soft law, or careful vs. blunt law, or good vs. bad law, or explicit vs. implicit law, or rule of law vs. rule of man (vs. rule by machine).
The option you will not have, not for very long, is no law. If you ban hard you get soft, if you punish explicit you get implicit, if you defeat careful you get blunt, if you fight good you end up with bad. If rule of law is unworkable, you have two options left, which one is it going to be?
Without Chevron, and with certain people fighting tooth and nail against any attempt to do precise well-considered interventions and also the general failures of Congress, there is less room (as I understand it) for improvised ‘medium’ solutions, and the solution types we would all prefer seem more likely to be blocked.
Thus I fear by default Adam is right on this on the margin. That also means that those most vulnerable to government soft power have to tiptoe around such threats, and those less vulnerable have no idea how to comply and instead hope they don’t trigger the hammer, which is not the way to do things safely.
My default guess is that things do not change so much. Yes, it will be a mess in many ways, but all the talk of big disasters and opportunities will prove overblown. That is usually the safe default. As I understand the ruling, you can still delegate authority, the only difference is that Congress has to explicitly do that. Mostly I’d presume various workarounds mostly suffice.
Deb Raji disagrees and sees this as gutting our ability to respond because we were entirely dependent on rulemaking authority, and the flexibility to respond as circumstances change.
Balaji of course calls this ‘Chevron Dominance’ and says ‘technology is about to accelerate.’ It’s funny. He thinks ‘Congress did not give the SEC the authority to relegate crypto’ as if being on a blockchain should make you immune to existing laws. The SEC has authority over securities. You made new securities. That’s on you. But more generally, he is saying ‘regulators just got disarmed’ and that everyone’s now free to do what they want. ‘I can already feel the T-levels across tech increasing,’ he says.
As another example, Austen Allred has a thread saying this ‘may be the most impactful thing to happen to startups in a long time,’ full of some very choice words for Chevron and the SEC. At some point that counts as supreme restraint. And certainly not being told how to comply with the law is infuriating.
I notice a clear pattern. For some people, no matter what It might be, It is always A Big Deal. Any little movement changes everything. Miami bans lab-grown meat? RIP Miami. California says giant frontier models have to do paperwork? RIP startup ecosystem. And it works in the other direction, too, Chevron is gone so LFG. They talk about lots of other aspects of a business the same way.
Scott Adams explained back in 2016 why Trump talks this way, it exerts maximum leverage until and unless people properly adjust for it. Similarly, everyone in crypto is always super hyped about whatever it is, and how it is changing everything. Which it isn’t.
Justin Slaughter thinks this is a sea change. You won’t be able to extend your authority to new areas as they arise without Congress approving, an increasingly tough ask. And he also warns of the shift to enforcement actions.
Justin Slaughter: Last year, on vacation with a friend who is very against crypto & senior in government, I asked him why the SEC wouldn’t just do regulations on crypto instead of enforcement. He said “it’s much easier for this Supreme Court to strike down regulations than enforcement actions.”
In the short term, I suspect a lot of agencies will take the Court literally rather than seriously and try to shift quasi-regulatory efforts on novel topics like crypto and AI into enforcement actions. @tphillips has some very thoughtful ideas on this.
I think it probably won’t work because this Supreme Court is very hostile to administrative powers that aren’t explicitly delegated. They’re trying to cabin all novel approaches.
When everyone says ‘oh great, now they will have to tell us the rules or else let us build, we can do all sorts of cool startups now!’ I sincerely hope that it works that way. I fear that in practice it is the other way. For crypto in particular I think the SEC is on solid ground from a technical legal perspective, and people should not get overexcited.
Critical Bureaucracy Theory: Privately, re Chevron Deference. I’ve seen quite a few tech entrepreneurs say this:
Generic Tech Entrepreneur: I think the impact of this may be disproportionately significant for start-ups. There are trade-offs when seeking guidance on what are legal / regulatory requirements when doing tech or business model innovation from agencies versus courts, but in my experience as an entrepreneur, legal precedent usually provides much greater certainty than “what will regulators decide about this three years from now after we’ve sunk lots of VC and three years of our lives into the business?”.
When you have fewer than, say, several thousand employees, it’s almost impossible to get a regulator to tell you anything or provide any kind of safe harbor statement until Megacorp forces them to act — obviously usually in a way that benefits Megacorp.
Matthew Zeitlin: One thing that lots of tech people genuinely believe is that they should be able to get advisory opinions and thus safe harbor from regulators and even prosecutors on their products and business practices and that they can’t is a great offense against the rule of law.
Houziren: Lots of people in general believe that the government should enunciate what the law is, and that fact that you never know you’ve broken the law until you’re found guilty really is a great offense.
Matthew Zeitlin: yes i agree that many people can’t think more than one step ahead
Matt Bruenig: Even during Chevron, the process of promulgating a rule was so insane and got so little actual deference from courts that for an agency like the NLRB for instance, it made far more sense to just signal possible law changes and decide adjudications than clearly lay out the rules.
The NLRB spent multiple years ticking off all the boxes for creating a formal regulation defining what a joint employer is for the purposes of the NLRA only to have a conservative district court judge in Texas zap it immediately. Why bother!
Anyways, the same procedural tricks that are being used to make regulating impossible (ostensibly for conservative political goals) also generate counter-strategies that make legal certainty impossible (which people say is bad for business!)
Matthew Anderson: The IRS does this too; but they are also willing to issue advisory opinions.
I agree we should aspire to what the tech people want here. We should demand it, to the extent possible, that we be told what is legal and what is illegal.
That is not, alas, how our system works, or how it fully can work. The regulators are not there to decide in advance exactly what the rule is for you.
In particular, they are not there to help you tippy-toe up to the edge, figure out exactly how to pull off your regulatory arbitrage, and then stand there powerless to do anything because technically they said what you are doing was acceptable and you don’t have to play by the same rules as Megacorp. Or, alternatively, to give you an opinion, then you use that to sue them. Also no fun from their side.
The good news from that perspective is this sets off a bunch of lawsuits. Those lawsuits provide clarity. The bad news is that this discourages rule making in favor of vague indications and case by case policy. That is not what startups want.
The Week in Audio
Carl Shulman spends over four hours on 80,000 hours talking about the economy and national security after AGI, and it is only part 1. A lot of the content is similar to Carl’s talk with Dwarkesh Patel last year.
I continue to feel like Carl is spending a lot of time on, maybe not the wrong questions, but not the questions where I have uncertainty.
Yes, there is a ton of energy available and in some theoretical sense we could do all the things. Yes, replication can if done efficiently happen fast. Yes, AGI could solve robots and do all the things. We know all that. The vision is ‘if we have lots of super capable AIs that do things humans want and coordinate to do that in ways that are good for humans, we would have all the things and solve so many problems,’ and yeah, fine, we agree.
Indeed, the central theme of this podcast is ‘people have this objection, but actually if you look at the physical situation and logic behind it, that objection matters little or is rather dumb’ and indeed, Carl is basically always right about that, most of the objections people make are dumb. They are various forms of denying the premise in ways more basic than where Carl ignores the implications of the premise.
They first goes through six core objections to Carl’s vision.
Why aren’t we seeing more economic growth today? Because we would not expect to until later, that is how exponentials work and the things that allow this rapid growth aren’t here yet.
How could doubling times be so much shorter than has ever been true historically? Because the historic doubling times are the result of physical constraints that will not apply.
Won’t we see declining returns to intelligence? No, we won’t, but also Carl points out that his model does not require it.
Indeed, I would say his model feels impossible to me not because it is so out there, but because he is assuming normality where he shouldn’t, and this is one of the key places for that. It is a vision of AGI without ASI, and he correctly points out there would be a lot of economic growth, but also there would be ASI. If you are pointing out repeatedly ‘doesn’t sleep, intense motivation’ and so on to contrast with the humans, you are not wrong and maybe people need to hear that, but you are missing the point?
Isn’t this an unrealistic amount of transformation of physical space? No, we’ve done it before and with AGI we would be able to do it again. Yes, some places might make that illegal, if so the action happens elsewhere. The places that refuse get left behind.
Won’t we demand more safety and security? He basically says we might want it but good luck coordinating to get it in the face of how valuable this stuff is on various fronts including for military power. No one is going to forego the next industrial revolution and be worth worrying about after they do.
Isn’t this all completely whack? Cool story, bro? No, not really, there are plenty of precedents, things not changing quickly would actually be the weird outcome. And it doesn’t matter how it sounds to you, previous tech revolutions sounded similar, what matters is what physically causes what.
So I indeed find those objections unconvincing. But the obvious seventh objection is missing: Won’t these AGIs very quickly have control over the future? Why would all this energy get spent in ways that benefit humans, even if you do ‘solve alignment’? And what makes you think you can solve that while charging forward?
I can’t get past this implicit (and often explicit) idea that something has to go actively wrong for things to end badly. The ‘risk of accidental trouble, things like a rogue AI takeover,’ instead of thinking that in a world transformed every few months where AIs do all the work and are more capable and efficient than us in every way us staying in charge seems pretty unlikely and weird and hard to pull off.
In the discussion of inequality and income, Carl says there will be tons of pressure from people to redistribute some of this vastly greater wealth, and plenty to go around, so there is no need to worry. Why would we assume this pressure impacts what happens? What is this ‘in democracies’? Why should we expect such things to long endure in these scenarios? Again, aren’t we assuming some very weirdly narrow range of AGI capabilities but not further capabilities for any of this to make sense?
The discussion of economists starts with Carl agreeing that ‘they say no way’ and yeah, they say that.
Then he goes over Baumol effect arguments, which are dumb because these AGIs can do all the things, and even if they can’t you can change the basket to work around the missing elements.
Or they deny robots can exist because robotics is unsolvable, which means they should not interrupt the people solving it, and also Carl points out so what, it would ultimately change little and not slow things down that much even if robots was indeed unsolvable because literal physical humans could be the robots with AIs directing them. And that’s largely good enough, because this whole scenario is actually being highly unimaginative.
What about input shortages especially for semiconductors? Carl answers historically rapid growth is common. I would add that with AGI help on this front too it would get a lot easier to go faster.
Carl points out that standard economic models actually very much do imply super rapid economic growth in these spots. Economists mostly refuse to admit this and instead construct these models where AI is only this narrow thing that does particular narrow tasks and make the assumptions that drive their absurd conclusions.
Won’t we be slow to hand over decision making to AIs? Carl points out that if the incentives are strong enough, we will not be that slow.
Why are economists dropping this ball so badly? They speculate about that, Carl points out some Econ 101 standard intuitions that stand in the way, and they are used to bold claims like this being wrong. And the economists expect everything to be gradual and ‘economic normal,’ and don’t get that this won’t hold.
They then spend an hour on the moral status of AIs. It is so weird to build up this whole model assuming the humans stay in charge, only then to notice that 99.999% of the intelligences in this world, that are more capable than humans, are not humans and may have moral standing, and then offhand say ‘well in these scenarios we have solved alignment and interpretability, so…’. And then they talk about these minds having open ended goals and wanting to survive and taking on risk and so on, and yes during this hour they notice the possibility of AI ‘domination.’
There is a part 2 coming, and it looks like it will address these issues a nonzero amount, but not obviously all that much.
I continue to find the Carl Shulman vision alienating, a weird kind of middle ground and way of thinking and doing math. Is it convincing to some people, as a kind of existence proof? I have no idea.
Bill Gatespredicts computer interfaces will become agent driven, but far more importantly that ASI is coming and there is no way to slow it down. He sees scaling as only having ‘two more cranks,’ video data and synthetic data, but expects success via improved metacognition that is more humanlike.
Oliver Habryka: I am confident, on the basis of private information I can’t share, that Anthropic has asked employees to sign similar non-disparagement agreements that are covered by non-disclosure agreements as OpenAI did.
Or to put things into more plain terms:
I am confident that Anthropic has offered at least one employee significant financial incentive to promise to never say anything bad about Anthropic, or anything that might negatively affects its business, and to never tell anyone about their commitment to do so.
I am not aware of Anthropic doing anything like withholding vested equity the way OpenAI did, though I think the effect on discourse is similarly bad.
I of course think this is quite sad and a bad thing for a leading AI capability company to do, especially one that bills itself on being held accountable by its employees and that claims to prioritize safety in its plans.
At least one person in position to know has said no such agreement was ever offered to them, so this was at least not universal. We do not know how common it has been.
Lawrence Lessig: You basically have a bomb that you’re making available for free, and you don’t have any way to defuse it necessarily.
We ought to be anxious about how, in fact, [AI] could be deployed or used, especially when we don’t really understand how it could be misused.
It’s not inconsistent to recognise at some point, the risks here need to be handled in a different kind of way … The fact that we believe in GNU Linux doesn’t mean that we have to believe in every single risk being open to the world to exploit.
Shakeel Hashim: Lessig, who is now a professor at Harvard Law School and representing a group of OpenAI whistleblowers, dismissed comparisons to previous technologies, where access to program code is considered to have improved security and fostered innovation. “It’s just an obviously fallacious argument,” he said. “We didn’t do that with nuclear weapons: we didn’t say ‘the way to protect the world from nuclear annihilation is to give every country nuclear bombs.’”
They train the model to spot mistakes in code. It finds mistakes more efficiently than untrained GPT-4 and better than human evaluators. For now, a human-LLM combined team does better still by reducing false positives.
They partly used intentional tampering to introduce subtle bugs.
Our goal is to find a method that will apply to long-form and open-ended tasks for which we do not have a ground-truth reward function. One could simply train critique models on unmodified answers but that approach has at least the following issues:
• Preference rates are impacted by a number of stylistic factors and may over-estimate model performance.
• Contractors may struggle to validate the correctness of free-form critiques if they make obscure claims.
• Contractors may struggle to spot important issues that critiques miss.
• Many answers will not contain severe issues, reducing the value of the data for improving critiques.
In addition to RLHF they use a technique called Force Sampling Beam Search (FSBS).
The critic also performed ‘out of sample’ on non-code examples, where it often managed to spot issues with samples previously rated by humans as flawless, sufficiently important that the humans no longer considered the samples flawless.
The conclusion is worth quoting:
Large language models have already passed the point at which typical humans can consistently evaluate their output without help. This has been evident since demonstrations of their strong performance on PhD-level science questions, among other impressive feats [25]. The need for scalable oversight, broadly construed as methods that can help humans to correctly evaluate model output, is stronger than ever.
Whether or not RLHF maintains its dominant status as the primary means by which LLMs are post-trained into useful assistants, we will still need to answer the question of whether particular model outputs are trustworthy. Here we take a very direct approach: training models that help humans to evaluate models.
These LLM critics now succeed in catching bugs in real-world data, and even accessible LLM baselines like ChatGPT have significant potential to assist human annotators.
From this point on the intelligence of LLMs and LLM critics will only continue to improve. Human intelligence will not.
It is therefore essential to find scalable methods that ensure that we reward the right behaviors in our AI systems even as they become much smarter than us. We find LLM critics to be a promising start.
Jan Leike, who contributed to this paper while still at OpenAI, offers thoughts here.
As a practical matter this all seems neat and helpful. The average accuracy of the evaluations will go up relative to human evaluations.
Code is easy mode, since the answer of whether it works is relatively objective. Value here is not so fragile. It is a good place to start. It also masks the dangers.
My concern is that this creates great temptation to rely on AI evaluations of AI, and to iterate repeatedly on those evaluations. It risks enshrining systematic correlated error, and amplifying those issues over time as the process feeds back upon itself. There are any number of ways that can go horribly wrong, starting with supercharged versions of all the usual Goodhart’s Law problems.
The average scoring, including the average human spot check, will look good for as long as we can understand what is going on, if we execute on this reasonably. Performance will genuinely be better at first. That will add to the temptation. Then the results will increasingly diverge.
AK: Self-Play Preference Optimization for Language Model Alignment
Traditional reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences.
Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this paper, we propose a self-play-based method for language.
Davidad: I think this is the new SotA prosaic-LLM-alignment post-training algorithm, besting DPO.
I do like the idea of working with preference probabilities. I worry about working self-play into the picture, as it seems likely to exacerbate our Goodhart’s Law issues.
Joshua Achiam: AI risk increases smoothly over time, in concert with capabilities, rather than discontinuously. But at some point the world will pass a critical threshold where we would lose a war against an AI adversary if such a war arose and the human side were unaided/unaugmented.
I am a little surprised, in general, at how underdeveloped the thinking is around what this conflict might look like if it happened. This seems like it should be at the root of a lot of threat modeling.
Several distinct things are usefully wrong here.
A few thoughts.
Our estimate of the path of future AI existential risk over time is changing like any good Bayesian estimate. Some events or information make the risk go up, some make it go down. Some insights make our estimate go up or down by revealing what was already true, others represent choices made by people.
Eventually, yes, the risk in the short term (~1 year or less let’s say), either of the event happening or us passing a ‘point of no return’ where in practice we are incapable of responding, starts to go up. From an outside view that may look steady, from an inside view it probably involves one or more large step changes as well, on key private and public decisions or on passage of time to critical points.
Top ten obvious examples after five minutes of thinking:
The decision to continue training, continue testing or releasing a new model.
A rogue actor decides to intentionally train and deploy an AI in a particular way.
A key secret, including model weights, is stolen and falls into the wrong hands.
The decision whether to institute key international cooperation or regulation.
A battle for control of a key institution, including both labs and governments
A catastrophic event or other warning sign that forces a response.
A war or other crisis even if caused by humans.
Discovery of a key new idea in capabilities or alignment.
An AGI/ASI gains the capability to successfully take control.
AGI/ASI becomes too central to our economy and discourse to dare act against it.
Some of these could be gradual, but many are likely or inherently sudden.
In particular, tie in the ability to take control versus the risk of it happening.
The traditional Yudkowsky or sharp left turn scenario is that these are the same thing. The highly intelligent and capable AI is going to attempt to take control if and only if it is confident that attempt would succeed at letting it fulfill its objectives (or it might well work and the risks of waiting are greater). The logic is obvious, and humans do their best to follow that logic as well.
Then there is the idea of a battle between ‘an AI adversary’ and ‘the human side.’
We hopefully have learned by now that there is no human side. There are only a bunch of humans, doing things. Their ability to cooperate and coordinate is sufficiently limited that our candidates in 2024 are Biden and Trump and we continue to race to AGI.
In the scenario in question, if the fight was somehow close and non-trivial, the AGI would presumably use various techniques to ensure there very much was not a human side, and many or most people did not appreciate what was happening, and many actively backed the AI.
The human side being ‘unaided/unaugmented’ is similarly bizarre. If the AI is sufficiently strong that it can take over all the systems that might aid or augment us, then I presume it is already over.
Why is this conflict not gamed out more?
Because there are mostly two groups of people here.
People who understand, as Joshua does, that at some point the AI will win.
People who will come up with any rationalizations as needed to deny this.They will come up with various increasingly absurd excuses and hopium as needed.
When someone in group #1 talks to someone in group #2, the goal is to convince people to accept the obvious. So you don’t game out exactly how the conflict works in practice or what the threshold is. You instead see what their absurd excuse or hopium is, and shoot it down and overwhelm it, and then they adjust, and you do it again. Occasionally this works and they become enlightened. When that happens, you are happy, great talk, but you are not closer to figuring out where the thresholds are.
When people in group #1 talk to each other about this, they still have radically different assumptions about among other things which AIs are against you and threat vectors and what scenarios might look like and how various things would work or people would react, and also the real scenarios involve effectively smarter things than you and also the details depend on unknown things about the future path of capabilities and conditions. So it is still super hard to make progress. And responding to a particular scenario on the margin based on how you think the battle would go is unlikely to turn losses into wins.
Mostly my answer is ‘yes, if capabilities do not stall we will effectively pass this point.’
Other People Are Not As Worried About AI Killing Everyone
From last week in audio: Aravind Srinivas, CEO of Perplexity, played a jarring mix of great founder and idiot disaster monkey on Lex Fridman. The parts where he describes the practical business of Perplexity are great, assuming he is not making things up. Then he will speculate about a future full of powerful AI agents doing everything, and say ‘I am not worried about AIs taking over’ as a throwaway line and get back to talking about other things, or say that open sourcing is the way to go because most people won’t have enough compute to do anything dangerous with the models.
I suspect that when Aravind says not worried, he and many others mean that literally.
Or as in the way most people find a way to not worry about death.
It is not that Aravind thinks this will not happen. We all know that the planetary death rate is holding steady at 100%, but what is the point of going all existential angst about it? If AI is likely to get us all killed somewhat faster this round, well, that’s unfortunate but in the meantime let’s go build one of those great companies and worry about it later.
He then combines this with failure to feel the AGI. He is super excited for exactly the AIs that he expects, which will be able to be creative like Einstein, do tons of that thinking without humans present and come back to you, act as your agents, and do all the other cool things, exactly enough to be maximally awesome for humans, but not so much that humans have to worry about loss of control.
How is that possible? Is there even a narrow window of theoretical capability where you can have those abilities without the dangers? I mean, no, obviously there isn’t, but you can sort of pretend that there is and then also assume we will stabilize in exactly that part of the curve despite then discovering all of physics and so on.
The good news is that running Perplexity is almost entirely about being a great founder, so in practice what he does is mostly good. The ‘answer engine’ idea is great, and occasionally I find it the right tool for the right job although mostly I end up at either the Google Search or Claude Sonnet ends of the spectrum.
Ab Homine Deus: Saying “I don’t believe in ASI” is just the most insane cope. Let’s say Einstein-level intelligence truly is some sort of universal intelligence speed limit. What do you think 1000s of Einstein’s thinking together thousands of times faster than humanly possible looks like?
1
1
*The following incantation is EXTREMELY powerful, please wield responsibly <3* 
