Listen on:
A few weeks ago, @Nathan_Barnard published What we can learn from stress testing for AI regulation. I had him on my new (and very informal) podcast to discuss.
I (Aaron) really think there's some insight here - more than his post's current karma count would suggest.
The podcast in general, including this episode, should be thought of as a "recorded conversation" more than an interview for which I specifically prepared. Nathan knows his shit though. It is earnest but unplanned and minimally edited.
Nathan elaborates on the concept of stress tests conducted by central banks. These tests assess the resilience of banks to severe economic downturns and the potential for a domino effect if one bank fails. They believe that lessons from this process can be applied to AI regulation. Aaron agrees, but also highlights the need for a proactive approach to AI regulation, as opposed to the reactive measures often seen in banking regulation.
Nathan suggests that institutions structured like central banks, staffed with technical experts and independent from government, could be beneficial for AI regulation. They believe such institutions could respond quickly and effectively to crises. However, they acknowledge that this approach may not be effective if AI development leads to rapid, uncontrollable self-improvement.
The conversation then shifts to compute governance, which Nathan sees as a promising area for AI regulation due to the obviousness of someone using large amounts of compute. They believe that this could provide governments with a control lever over cutting-edge AI labs, similar to how central banks control banking loans and affairs.
Nathan acknowledges that the leaders of major AI labs seem sensible and aligned with AI safety principles. However, they argue that regulation and public actors can play a crucial role in creating common knowledge between labs and preventing a race to the bottom. They also discuss the potential benefits and drawbacks of different regulatory approaches.
Nathan believes that post-crisis financial regulation, such as the Dodd-Frank Act, has generally been effective. They suggest that AI regulation could follow a similar path, especially if AI becomes a significant part of the economy. However, Aaron expresses skepticism about the ability of political processes to produce effective AI regulation.
The speakers agree that pre-crisis regulation has generally been less effective than post-crisis regulation. They discuss the potential for AI regulation to follow a similar pattern, with effective regulation emerging in response to a crisis.
The conversation concludes with a discussion on regulatory arbitrage, where banks shift activities to where it's cheapest to do business. Despite evidence of this behavior, Nathan notes that there was no race to the bottom in terms of regulation during the financial crisis.
Note: created for free by Assembly AI; very imperfect.
Also, the first few minutes got cut due to technical difficulties, so it sounds like we start in the middle of our conversation.
AARON
I guess one thing is like, okay, so with the fed, I honestly don't know how the mechanics of, for example, federal whatever the funds rate.
NATHAN
The federal funds rate?
AARON
Yeah, there's like various things like that. Okay, so somebody like Jerome Powell signs a piece of paper and it says the federal funds rate off by a little bit. Okay. I don't know how the mechanics works after that. One general impression I have, and you can correct me obviously the US federal government at large just has a bunch of stop and go levers within the banking system such that if the US wants to basically stop most commercial banking lending pretty quickly, which obviously wouldn't want to do, but if it did, it just could or something. Whereas right now I just own a computer and I can just run PyTorch on my computer and the military would have to break into my house to stop me.
NATHAN
Yeah, I think this is where Compute governance stuff seems like really important, at least in the sort of the current paradigm, having very large amounts of computers, like important for sort of trained state of the art models. And this is much more controllable. It's true. If you're in the world where just like anyone on the laptop could run safety art model, then I think you'd be in a lot of trouble. I think the bull case for compute governance as a way to do AI regulation is like, compute so amongst the three inputs into the AI production function, like computes data and algorithms, compute like a lot of choke points. It's obvious when someone's using a large amount of it basically made obvious when someone's making it of death, someone's using a large amount of it. And this makes it the most obvious lever by which governments could be able to have the control over cutting edge labs working on the cutting edge they do over the ECB and the banking loans and affairs and central bank.
AARON
Am I correct to assume that you listened to the 80k podcast with Leonart Heim?
NATHAN
No, I've read.
AARON
Oh, okay. Because I figure just because then I'm very impressed because you're hitting all the points you're like summarizing the podcast right now.
NATHAN
Okay.
AARON
So I would recommend that to my fellow uneducated people.
NATHAN
Yeah, I suppose as an aspiring AI governance researcher and this is the sort of professional knowledge which I'm sort of required to.
AARON
I'm glad. It would be bad if you guys were I mean, not going to lie. Sorry, dude. I kind of trust. I don't know, Leonard seems like a little bit more senior or experienced. Yeah, but I'm glad you guys are basically converging evidence.
NATHAN
This is from stuff I've read from read from within the community.
AARON
I'm not really converging in like an accident.
NATHAN
Yeah, I think sort of lots of people lots of people have been thinking about this for a long time. But yeah, learn time is much more seniors than -
AARON
Okay, so what's your-
You can also just carry on. I don't know if you had a thing, but your vision for, I don't know, compute governance.
NATHAN
I'm afraid I'm not a compute governance guy. This was more just a reference to how we could why compute governance is exciting and why sort of within if we sort of try to do air regulation, why compute governance seems like such like such like a promising thing to work on to make the full stack of air regulation sort of work.
AARON
Well, yeah, I guess. One thing, and I feel like a bad liberal saying this for extremely contingent reasons I think it seems plausible that four guys across some AI. Labs are just maybe not, in fact, responsible enough to have a really good shot at controlling their own, I guess self regulating AI. Or cutting edge AI. Or whatever. But it seems like an unusual situation in that plausibly just like a couple of people at the big labs would in fact do a better job than the outcome of whatever fucked up political process in the US. Generate. Like an agency.
NATHAN
Yeah, I think this is, like, a thing that we have to really grapple with. That Sam Altman does, in fact, seem like quite a nice guy and pretty sensible and has reasonably good alignment takes. And similarly for Dennis Hasabis, and similarly for Dario and Daniela Amadei do just in fact seem like quite sensible people and probably much more bought into much more air safety pills than I'd expect air existential safety pills than I'd expect a regulator to speak. I think this is just a thing you have to grapple with. Yes, I agree. Yeah, sorry, go ahead. I think over the longer run, how much churn do I expect there to be in like which leading company for safety art AI systems? I suppose if one has like five year timelines, then probably not very much chat, but if one has like 50 year timelines or even 30 year timelines, I think we should expect a lot more Chan and these worlds I'm more these sort of like medium to longer timelines worlds. Yeah, I think I'm much less confident that the sort of particular individuals leading custom edge AI labs will have alignment takes and making AI go well takes library with I'm more bullish on regulation in these worlds.
AARON
Yeah, I guess hopefully they don't necessarily compete with one another. I mean, maybe they kind of do, but seems like they can kind of stack, so hopefully we can have both.
NATHAN
Yeah, I think this would be good. I think there's also maybe another role that regulation and public access can play, like creating common knowledge between labs. None of them are engaging in race to the Bottom. It sort of allows them to way of them all binding their hands. Maybe this won't matter very much, but I think that could be another case for regulation. Even if you think that the specific takes of the various specific individuals with power leading AI labs are quite good.
AARON
Yeah, I don't know. I just get gloomy for, I think justified reasons when people talk about, oh yeah, here's the nine step process that has to take place and then maybe there's like a 20% chance that we'll be able to regulate AI effectively. I'm being facetious or exaggerating, something like that, but not by a gigantic amount.
NATHAN
I think this is pretty radically different to my mainline expectation.
AARON
What's your mainline expectation?
NATHAN
I suppose I expect like AI to come with an increasing importance past economy and to come up to really like a very large fraction of the economy before really crazy stuff starts happening and this world is going very anonymous. Anonymous, anonymous, anonymous. I know the word is it'd be very unusual if this extremely large sector economy which was impacted like a very large number of people's lives remains like broadly unregulated.
AARON
It'll be regulated, but just maybe in a stupid way.
NATHAN
Sure, yes, maybe in a stupid way. I suppose critically, do you expect the stupid way to be like too conservative or too like the specific question of AI accenture it's basically too conservative or too lenient or I just won't be able to interact with this.
AARON
I guess generally too lenient, but also mostly on a different axis where just like I don't actually know enough. I don't feel like I've read learned about various governance proposals to have a good object level take on this. But my broad prior is that there are just a lot of ways to for anything. There's a lot of ways to regulate something poorly. And the reason insofar as anything isn't regulated poorly it's because of a lot of trial and error.
NATHAN
Maybe.
AARON
I mean, there's probably exceptions, right? I don't know. Tax Americana is like maybe we didn't just kept winning wars starting with World War II. I guess just like maybe like a counterexample or something like that.
NATHAN
Yeah, I think I still mostly disagree with this. Oh, cool. Yeah. I suppose I see a much like broader spectrum between bad regulation and good regulation. I agree it's like very small amount. The space of optimal regulation is very small. But I think we have to hit that space for regulation to be helpful. Especially in this especially if you consider that if you sort of buy the AI extension safety risk then the downsides of it's not this quite fine balancing act between too much whether consumer protection and siphoning competition and cycling innovation too much. It's like trying to end this quite specific, very bad outcome which is maybe much worse than going somewhat slowing economic growth, at least somewhat particularly if we think we're going to get something. This is very explosive rates for economic growth really quite soon. And the cost of slowing down economic growth by weather even by quite a large percentage, very small compared to the cost of sort of an accidental catastrophe. I sort of think of Sony iconic growth as the main cost of main way regulation goes wrong currently.
AARON
I think in an actual sense that is correct. There's the question of like okay, Congress in the states like it's better than nothing. I'm glad it's not anarchy in terms of like I'm glad we have a legislature.
NATHAN
I'm also glad the United States.
AARON
How reasons responsive is Congress? I don't think reasons responsive enough to make it so that the first big law that gets passed insofar as there is one or if there is one is on the Pareto frontier trading off between economic growth and existential security. It's going to be way inside of that production frontier or whatever. It's going to suck on every action, maybe not every act but at least like some relevant actions.
NATHAN
Yeah that doesn't seem like obviously true to me. I think Dodd Frank was quite a good law.
AARON
That came after 2008, right?
NATHAN
Yeah correct. Yeah there you go. No, I agree. I'm not especially confident about doing regulation before there's some quite bad before there's a quite bad warning shot and yes, if we're in world where we have no warning shots and we're just like blindsided by everyone getting turned into everyone getting stripped their Athens within 3 seconds, this is not good. Both in law we do have one of those shots and I think Glass Seagull is good law. Not good law is a technical term. I think Glass Steagall was a good piece of legislation. I think Dodd Frank was a good piece of legislation. I think the 2008 Seamless Bill was good piece of legislation. I think the Troubled Assets Relief Program is a good piece of piece of legislation.
AARON
I recognize these terms and I know some of them and others I do not know the contents of.
NATHAN
Yeah so Glass-Steagall was the financial regulation passed in 1933 after Great Depression. The Tropical Asset Relief Program was passed in I think 2008, moved 2009 to help recapitalize banks. Dodd Frank was the sort of landmark post financial crisis piece of legislation passed in 2011. I think these are all good pieces of legislation now. I think like financial regulation is probably unusually good amongst US legislation. This is like a quite weak take, I guess. It's unusually.
AARON
So. I don't actually know the pre depression financial history at all but I feel like the more relevant comparison to the 21st century era is what was the regulatory regime in 1925 or something? I just don't know.
NATHAN
Yeah, I know a bit. I haven't read this stuff especially deeply and so I don't want to don't want to be so overcompensant here but sort of the core pieces which were sort of important for the sort of the Great Depression going very badly was yeah, no distinction between commercial banks and investment banks. Yes, such a bank could take much riskier. Much riskier. Things with like custom deposits than they could from 1933 until Glass-Steagall. And combine that with no deposit insurance and if you sort of have the combination of banks being able to do quite risky things with depositors money and no deposit insurance, this is quite dangerously known. And Glass-Steagall repeal.
AARON
I'm an expert in the sense that I have the Wikipedia page up. Well, yeah, there was a bunch of things. Basically. There's the first bank of the United States. There's the second bank of the United States. There's the free banking era. There was the era of national banks. Yada, yada, yada. It looks like 19. Seven was there was some panic. I vaguely remember this from like, AP US history, like seven years ago or.
NATHAN
Yes, I suppose in short, I sort of agree that the record of sort of non post crisis legislation is like, not very good, but I think record of post Cris legislation really, at least in the financial sector, really is quite good. I'm sure lots of people disagree with this, but this is my take.
AARON
Yeah. Now I feel like this is probably - I have no idea if this is productive direction, but feel like maybe there's almost something resembling anthropics going on, which is like if you hadn't had effective financial regulation, we wouldn't be talking about transformative AI. The economy that supports the development of transformative AI depends on a strong economy that's robust to banking failure. And so there's like anthropic effects going on.
NATHAN
So I suppose one is I'm like, I don't know, quite skeptical of bringing up anthropics in these kinds of areas. We then got quite bad financial well, the US got quite bad financial regulation from the mid 90s until the this result in 2008 financial Cris guess when.
AARON
The golden era of deep learning wasn't during that time.
NATHAN
Sorry. The US did in fact have quite bad financial regulation for 20 year period.
AARON
Right. What I'm saying is the era of transformative AI was not during the period either during or like directly following, like this period of bad regulation.
NATHAN
I think that's, like, got nothing to do with like USA. Khan was doing very well in 2005 and 1997. Okay.
AARON
This is like a very speculative, off the cuff thing. Yeah. Don't necessarily endorse anything I said in.
NATHAN
The last.
AARON
Yeah, no, I don't know. I feel like maybe I'm just also being I really just don't know. I'm like slinging takes based on broad impression. We love some takes, but I basically defer to the median EA ish or I guess not ish EA governance researcher?
NATHAN
No, there's some selection effects and governance researchers are selected people who think governance has a chance of working. I saw how strong a selection effect is. Probably quite strong. I think it's really worth pushing back on.
AARON
You're such a good representative of the effective altruism community.
NATHAN
I endeavor to be.
AARON
Sorry. Okay. Is there anything else more to talk about on this matter?
NATHAN
Oh, many, many things.
AARON
You mean we didn't fully solve compute governance just now?
NATHAN
Many things. Just on the financial regulation case studies.
AARON
Oh, right, okay.
NATHAN
Yes, keep going. Might have exhausted people's appetites, Pigeon Hour listeners.
AARON
Fuck them if I get for it, I guess stop you, but I guess keep explaining the analogy.
NATHAN
Cool. I think the analogy is not terribly weak, but I don't want to sort of overstate how strong I think the allergy is. I think most of the value here just comes from looking at just looking at complex regulation, how complex regulation works. I think maybe the second key takeaway from looking at stress tests in particular is there was not any race to Blossom Dynamics, despite the fact that there is I think really quite robust evidence that banks do engage in regulatory arbitrage. It's like very hard to cause an inference it's very hard to cause an inference on whether banks do engage regularcy arbitrage.
AARON
Can you explain regulatory yeah concept?
NATHAN
Yeah, just basically trying to shift assets to shift activities to where it's cheapest to do business.
AARON
Right?
NATHAN
This can either be like in terms of trying to assets like off balance sheet, so keeping assets whatever legal restriction they're in, but sort of shifting them being in a different part of the bank. So using the reef home markets instead of other ways of doing interbank borrowing, be one example of this, but also just more obviously moving assets out of jurisdictions, moving assets from US bank, moving assets from the US to its UK branches, for instance. So they come up with regulation.
AARON
I mean it's always oh yeah, Ireland.
NATHAN
Is Ireland, or you have quite boss evidence that the banks do do this. But despite this, there was no it's been both the case that basically all central banks, all central banks that matter have adopted really quite stringent stress testing regimes for financial crisis. Despite, it seems to me that their domestic financial system probably could have gained some advantage by banks not adopting these sort of the expense of the global financial system. And also at least in the literature I've read, in the interviews I've read, there's not any discussion of fear of regulatory arbitrage as something constraining the sorts of regulation which the central bank could adopt. I think this is a pretty interesting takeaway.
AARON
I mean how much of this is just because of correlated economies?
NATHAN
Not just wait.
AARON
So the theory behind why there would be a race to the bottom is that, for example, the UK makes it cheaper to do business in the UK. And banks move capital like you can imagine. On one end of the spectrum. It's like a perfectly global, like a super globalized economy. Like, like total, like free border, like, economically, like no borders or whatever.
NATHAN
Yeah.
AARON
In which case I'm pretty sure that reasoning falls apart right under this hypothetical perfectly globalized economy because everything equalized or the economic benefit just equalized throughout the world without respect to borders or whatever.
NATHAN
And then like, can you be more concrete?
AARON
Okay, yeah, that was like a terrible explanation. So, okay, like just like, imagine like two different states of the world, one of which is both of which are way more extreme than the actual world right now. And so on one hand, you can have perfectly closed economies like every country, all economic activity happens within their borders or whatever. And on the other extreme, you have a perfectly globalized economy which case rates.
NATHAN
The same across, like interest rates the same across all countries.
AARON
Sure, yes, interest rates and also, I guess, returns to everything.
NATHAN
Both wages and interest rates the same. Right.
AARON
You're good. Okay. Man, this guy's sharp. Okay.
NATHAN
Doing a British undergraduate degree. If you learn the fucking subjects you study.
AARON
In America, we get to study more than one thing.
NATHAN
Ha.
AARON
These are both like okay, in the libertarian paradise world.
NATHAN
Yeah.
AARON
So central banks only have, I'm very much like thinking and talking at the same time, as you can probably tell, but that's fine.
NATHAN
This is what writing hard, this is what question hour is for.
AARON
Exactly.
NATHAN
Writing hard is fucking easy.
AARON
Man. I'm realizing how poor of a grasp. I have no idea what a central bank does, and realistically, I think I kind of do. They do a bunch of things. I know the Fed, like, adjust the federal funds rate. They control the money supply. They're like the bank for banks above and beyond that, I have no idea.
NATHAN
So you also do fat regulation for our discussion. That's their key other function. They also do fat regulation.
AARON
Right? And so any financial regulation in this libertarian paradise pauses. It might adjust the interest rate. For example.
NATHAN
Everything should equivalent race, right?
AARON
So there's no race to the bottom because everybody's by definition or by assumption in the same.
NATHAN
There are no supernormal profits, everyone perfectly invested markets and so forth. Okay. I suppose the key way this is not like our current world is interest rates have in fact not equalized across countries. And also we empirically see at least again, I want to emphasize cause inference is very difficult in this area. But insofar as we can have cause inference here, as well as looking at sort of more basic measures, like much more basic regressions, we do in fact see that when financial regulation gets more stringent in country A, the banks in country A move assets to country B where financial regulation is less stringent.
AARON
Meta level aside, didn't you make a thread that's like all your rejections to EA places?
NATHAN
Oh, I did? Yeah. Okay. And also non EA places. Mostly non EA places.
AARON
Okay. I'm getting so black pilled on everything because you know your shit for a bunch of random things, like random important things. Not random, but important things. And so yeah, man, there's no hope for the rest of us anyway. Okay, back to the object level. Keep going. Yeah. Interest rates haven't normalized.
NATHAN
Yeah, I think you should be less blackpilled than just very dark gray. Well, we can come back to that later.
AARON
I mean, whatever you want. Yeah. Now or later.
NATHAN
Yeah, let's just finish off this thread and then we can maybe come back to that discussion. Yeah. Interest rates haven't equalized and we do see this empirical phenomenon of banks moving assets towards countries which cheaper do business. Assume that other country is also a high income country. I. E. Banks don't move from, like you can make your regulation make it as cheap to business as you want. In the know. Wells Fargo is not moving any of its assets to the Congo. And so in this context, I think it is, on its face, interesting I say, actually not surprising, but I think it is on its face, interesting that we don't see this race at the bottom between central banks in terms of, like yeah. In terms of post 2008 financial regulation. I think maybe the sort of analogy here to AI is that, yeah, we might have a reason to think this. I think maybe gives us some reason to think that international competition international competition is, like, less of a constraint on regulatory coordination, even though any explicit coordination between countries and without the US. Having to use its either soft or hard power to enforce it. So the Fed had played no part in forcing the ECB or the bank of England or the people of bank of China to adopt stress tests. I think this is actually quite encouraging for thinking about whether a lack of international coordination will doom AI regulation.
AARON
Yeah. At least for a while. For last 20 minutes or whatever. Lost track of what this analogy was for. No, but I think it's pretty direct. Right. So we're talking about country level regulation versus country level regulation on these two domains. And I guess my impression is that the financial systems across the developed world are much more equal than the AI infrastructure or, like, AI economies.
NATHAN
Yeah, I think that's very clearly true.
AARON
Yeah. Okay, so maybe this is just, like, I guess good, because you only have to worry about one government instead of.
NATHAN
Like at least I think if you sort of create a good model of regulation, which is what happens with stress tests, is that stress tests really proved their worth in 2009. They really restored confidence in the US banking system. If you have this good model regulation, then they can be then they sort of read the thing that will be adopted very widely, which will happen with the stress test in particular.
AARON
Yeah, cool. Yeah, sounds good.
NATHAN
Yeah. I think maybe come back to checking our stress tests some other time. I've got six more takeaways. No, I've got four more takeaways. That might be a little excessive.
AARON
You can at least list I don't know how much maybe we won't have like an hour for each or something. But you can do the short version at least. Maybe what are they? And then maybe we can figure out which ones to dive in.
NATHAN
Sure, let's find the doc. Let's find the doc. Here's the rest of the doc. Yeah, I think this is maybe the third and most other really big one is like, credit rating agencies, I think, play like a pretty major part in a financial crisis. Yeah, I think, yes, credit rating agencies play like a pretty major part for making the financial crisis. And I think at least some of the ways in which we're doing air regulation now seem to me to have similarities to how achieves were structured. I think that's maybe a second takeaway I think that the other takeaway is that industry standards just were very important in influencing what regulation was. I think it's been like a big question in thinking about how much sway do we think that interlab agreements currently, how much influence can we have on long term air regulation via getting interlab agreements? Interlab standards? I think the lesson from regulation, especially in the US. Over the past century or so is a lot, is it could matter a lot and could even be the most important. Think the final one, the third takeaway still haven't already chatted about is banks mostly haven't been able to gain the stress testing system. Those are the three we haven't sourced our.
AARON
I mean, these all seem like good analogies. Which one of these jumps out to you as the most insight, like insight rich?
NATHAN
Yeah, I think the industry standards one is maybe the industry standards one might be worth talking about. I think it really is striking just how just how influential financial industry, industry level financial regulation has been influencing the long trajectory of US. Financial regulation.
AARON
We're talking about the laws or non law agreements, norms within companies doing the yeah.
NATHAN
Various ways in which banks and other financial institutions have regulated themselves through non legal channels. I've had really profound influences on the legal ways in which they've been regulated.
AARON
Okay, no, I don't know anything about this and I'm kind of surprised and also okay. Yeah. So what are some of these norms?
NATHAN
Yeah, so stress tests are one the stress tests started out as something done within the, within the mortgage industry, and then they were adopted pretty wholesale. The methods of stress testing were adopted pre wholesale by central banks after financial crisis. There's not much more of a flashy takeaway here. This is just the takeaway. This is just the first takeaway. Just like this method which they used is now just used in a very similar way and it now has the force of law that's the first 1 second one, I think this is maybe the most striking is a credit rating agency sort of pops up in the US. In sort of the early part of 20th century.
AARON
I'm sorry, there's something with the audio. Could you say that again?
NATHAN
Yeah. So stress tests were first used by the mortgage industry in the sort of 1990s okay. And now have just been adopted by all the major central banks.
AARON
Okay. So first there's like inter industry proliferation and proliferation and now you've added international legal.
NATHAN
Yeah. And now it's just now it's just international legal proliferation. Yeah. Wow. Yeah, that's it. That's the takeaway there for that one. I think the second sort of really striking example is credit rating agencies. Credit agency sort of pops up in the early 20th century in the US. And went, right, we're going to rate bonds. And so by rating bonds, investors will know and some other mostly bonds will know how much they can trust them. And then both the specific organizations doing the credit rating and the credit ratings themselves became enshrined in US law in 1975. So a large number of financial institutions, like pension funds, I think most importantly are only allowed to invest in bonds rated prime or best in the prime category by credit rating agencies. And the only credit rating agencies which this isn't quite true anymore, but is pretty close to being true, the only credit racing agencies which count are with the private credit racing agencies established in the early part of the 20th century. Yes, I can be much more concrete. So, like whether the US teachers pension fund wants to buy some bonds and the US government says, cool, you can only buy bonds rated crime or better and rated crime or better by Moody's Pictures or Standard and pause, there are a few others, but basically just those.
AARON
Kind of sounds like just regulatory capture by Moody's. Do you think that's like the right read or is it actually like a wise I think it's like philosopher king regulation.
NATHAN
I think it's regulatory laziness.
AARON
Okay.
NATHAN
They were all very established by 1975 and it was just much easier to piggyback off them than discover their own.
AARON
Regulation object level though. Is it even good to for the government? I honestly don't have a form take at all. Whether it's good for whether you should expect the market to be able to handle risk appropriately or whether you need want regulations on what types of bonds like the teachers union, whatever can buy.
NATHAN
Yeah, I have no especially strong take here. My weak take is yes. My sort of takeaway from like my weak take is yes. I think there's actually reasons to think that this could have quite bad untended consequences because basically something quite similar did have quite bad untended consequences in the bad untended consequence was a financial crisis weekly. Yes. But the reason I weren't characterized as regularly captured by Moody's is like from looking a little bit history of this, it didn't seem to me there was lots of lobbying efforts by the big credit racing agencies to have themselves enshrined in law. Like this. And they're also quite small organizations. They're just like many times smaller both in terms of revenue and staff compared to the large banks. So I should be very surprised that lobbying power is able to compete with this is this is quite a weak tech. I haven't looked that deeply into the history of this.
AARON
I feel like I'm going to add you next to Carl Schulman on people I can call upon to have a take on a well informed take on arbitrary matters.
NATHAN
Sorry, I still missed that. Someone called sorry.
AARON
I'm mostly joking, but I said I'm going to adding you to my list of people that I can call upon to have a well informed take about any arbitrary matter.
NATHAN
That's very kind of you. Do bear in mind I have been working this full time for the last whatever you say, like three weeks or something.
AARON
Whatever you say. I bet you still know random shit about other stuff too. Anyway. Yeah. Okay. I feel like this generally I think I'm just generally more optimistic about coming back to AI about industry sourced yeah.
NATHAN
Regulation. I think I am as well. I basically think the current trajectory I think it's very plausible that ARC Evals gets enshrined into law.
AARON
That's wild. Sorry? It's wild. I don't know, maybe it's not wild. It's wild to me. I don't know. It's just like a little bit of like it makes total sense, actually. It sounds right. It's just like putting plus two other people in the same bucket as Moody's is just like I don't know, it's like a good illustrative vignette, I feel like, for the new AI era. Anyway, maybe I'm reading too much into this.
NATHAN
Yeah. I also haven't looked really deep. I've only looked into sort of financial regulation. I haven't looked into other industries here. So again, don't want to have too much force on any of these financial regulation.
AARON
And also, I feel like Paul always gets the credit. Paul's great. We know. There's also Beth Barnes and Mark.
NATHAN
Yeah. Beth Barnes is awesome.
AARON
We'll give them a shot. Okay. I'm probably missing more people I'm sorry, other arch people on this podcast that gets dozens of downloads.
NATHAN
Dozens of downloads.
AARON
Dozens. Yes. Okay.
NATHAN
You just want to wrap it up. Maybe you just want to wrap us up. So just one more take. Is it banks mostly haven't been able to game the system, have those haven't been able to gain the stress game the stress testing system? I think they have a bit, but how I take both from qualitative work interviews with individuals working at individuals working at banks and from looking at the status of whether larger banks be able to large banks been able to sort of get around regulation in various ways. For instance, by sort of not having to recapitalize themselves when other banks would other sort and less powerful banks would have. Yeah, it seems like banks mostly having to gain the system and it's sort of not just a tick box exercise and pathways is that the Fed, BCB, and the bank of England's models all private. The banks don't know them. And also the specific scenarios change year to year. I think this is not that important. I think it's mostly telling us that, yes, even institutions as powerful as the Fed, the ECB and bank of England, even if Face, is very powerful and very sophisticated actors, probably the most powerful sophisticated actors in the world economy, the large banks are able to effectively use their regulatory power to constrain them. I think that's the takeaway from here, mostly not entirely, but I think mostly.
AARON
This is actually pretty surprising. Yeah. I feel like if this wasn't the case, we would be seeing more financial crises. But it's like object level. There's so much incentive to basically just game a metric and I don't know, usually when there's heavy incentives to game a metric, people are pretty good at gaming the metric.
NATHAN
Yeah. I think you shouldn't update too much on this because financial crises, financial crises, like large financial crises are, like, structurally rare. So I don't want to sort yeah, I think we don't have that much evidence just from looking at base rates of financial crises to see to draw from here. Okay. So I wouldn't update too much on this. Yeah. I think the qualitative evidence, I think here looks like quite strong, as in just when you sort of interview bankers, they sort of say it isn't just a tick box exercise. And yeah, so, secondarily, just like very large banks have been forced to not pay out dividends and not do share buybacks to recapitalize. Like Citibank. In 2014, their share price dropped 6% when it was found they failed the stress test. This is like a serious financial penalty. This is like a no fucking.
AARON
Yeah, that's probably, what, at least tens of billions of dollars in market cap. Yeah.
NATHAN
And they just weren't allowed to pay dividends for that.
AARON
I didn't even know that was like.
NATHAN
A thing that's like a no fucking about test.
AARON
Yeah. Okay.
NATHAN
So I suppose maybe, just maybe, I just want to sum up on how I'm feeling about how this is very broadly updated me in terms of how I'm sort of feeling about air regulation. I think it's made me think that in worlds we don't have warning shots, I'm a bit more scared and even quite a lot more scared. It really does seem like a financial cris was really quite critical and the great question for that really was quite critical in getting better regulation. I think that's, like, the major Pessimistic update from this. I think the major optimistic update from this is that if you have powerful institutions like Central Bank, they can do really good regulation. What I judge is really quite good regulation against the most powerful actors and the various use the sort of rationalist term like Molochian forces do not constrain them here.
AARON
Yeah. And if anything, I feel like things, at least in this particular domain, there's various factors that actually push in the optimistic direction, like individuals being alignment pilled, whereas bankers are maybe not banking alignment pilled.
NATHAN
Yeah, I think that's basically right. I think that's basically right.
AARON
Okay, cool. So we will resume at some point in the future.
NATHAN
All right? Okay. Awesome. Aaron, thanks for having me on. On.
AARON
Thank you.