I wasn't trying to say that you had provided no argument for it, sorry! I was just curious whether you had written about this previously with a handy link. It feels like a theme in a bunch of your writing, but you seemed in a better position to remember any specific essay or section. 

If you propose a well-operationalized bet and an adjudicator and escrow arrangement I will accept or reject the proposal.

I'll think about it over the next day or two and see whether I can find something. I am currently skeptical we can find something given that I don't expect shifts at the scale of "Stanford stop being a top university at all". But I'll try for a bit.

The problem is that Alameda then used that FTT as collateral for other loans from external parties, while of course presenting the FTT as an uncorrelated asset!

Caroline said directly: 

We were borrowing billions of dollars from Genesis using FTT as collateral on our loans,

I agree that if Alameda had just borrowed from FTX, then only FTX would have been fooled. But given that Alameda loaned billions of dollars from Genesis using the FTT as collateral means external parties were fooled in the way I am trying to describe in this post!

I am pretty sure Michael thought he was largely agreeing with me. He wasn't saying "I agree this thing is important, but here is this totally other thing that I actually think is more important". He said (and meant to say) "I agree this thing is important, and here is a slightly different spin on it". Feel free to ask him!

intrinsic pride in the work, intellectual curiosity

I mean, both of these seem like they will be more easily achieved by helping build more powerful AI systems than by building good tooling for alignment research. 

Like I am not saying we can't tolerate any diversity in why people want to work on AI Alignment, but like, this is an early career training program with no accountability. Selecting and cultivating motivation is by far the best steering tool we have! We should expect that if we ignore it, people will largely follow incentive gradients, or do kind of random things by our lights.

I am sorry, I don't understand. Your bullet points says nothing about corrigibility. It says that Bostrom said that AIs will not understand what we mean. I mean, feel free to edit it, but I don't understand how what you are saying is responsive to what I said.

It's true there is a bit of a tension. Some thoughts: 

• Specialization is IMO really not the key component of most assembly lines. I am mostly familiar with automated assembly lines, where maintenance workers tend to be generalists and work on a wide variety of machines, but I expect even in largely human-driven assembly lines, the time it takes to learn a new station tends to be on the order of days (given basic proficiency in the kind of thing that the factory is producing), and you continuously try to make the task be doable by more people.
• A thing that is a much bigger aspect of assembly lines is "focus", which plays out on the scale of hours, not at the scale of weeks. Switching tasks requires switching cognitive contexts. Being given a station for a day means you can focus on a single task. But in order to optimize how you perform that task at a high level you probably want to work many stations on different days so you can see how work at your station affects the work at other stations!
• When I am imagining the workshop snake, it's not super clear how much benefit you get from specialization, though maybe! I am imagining most stations do not involve humans at all, but instead involve doing exercises or reading something. Humans are not great implements in a teaching pipeline. You will definitely need them sometimes, but you were probably imagining a setup with many more humans than I was.

Beyond that, it's great if you do end up with a process that produces value by creating copies of the exact same widget, that take the exact same time, and if you do that, you can tolerate a lot of specialization, but of course things rarely work out that way. Especially in anything software adjacent, where you have software handle the uniform parts and so the labor you are trying to optimize is left with the heterogenous parts. 

So what do you do if different stations randomly take longer or shorter, and your work product is very non-uniform and so requires different amounts of inputs from different stations each time? You need to find ways to transfer capacity from different stations on the assembly lines to other stations on the assembly line. This is what generalist labor is about. Generalist labor provides the slack in the system that allows it to maintain high throughput and efficiency. 

At Lightcone, the way this plays out is that our core team is a lot of generalists, and then when we do end up having really locked down what a task is like and where it fits, we do hire labor to actually perform it every day who don't need to be generalists. Lighthaven in terms of maintenance and restocking and cleaning is not done operated by generalists, we hire contractors and specialized staff for that.

Another thing I maybe under-emphasized in the post is that the key criterion you want to optimize for is total production time. The whole point of single piece flow and small batches is to maximize the speed at which you get feedback about the consequences of your actions. The depth of your assembly line is a cost to that! Every time your assembly line gets longer, you have to wait longer to see the consequences on the final product.

Now, that's a big of high level on the tension here, but I realized I didn't really explain a good model of why narrow specialization in a non-uniform work environment pushes heavily towards big batches and waterfall planning. 

Let's say you have an assembly line with 4 stations (A, B, C, and D), with a lot of variance at each station about how long work at that station takes. Let's talk about a few different scenarios: 

1. You have 4 employees each capable of only performing the job of one station.

Now, let's say station A takes twice as long this time. This means station B, C and D will now be idle 50% of the time. The total efficiency of your process is $\frac{1}{2}$. 

[Ok, I have to run, but I'll edit this comment with the full explanation later, though maybe it's already clear]

As I understand, it stole mainly to cover the trading losses and not to pay for advertising. If it had accepted the trading losses and wound down the trading, it wouldn't have looked very different from the outside.

Advertising spending ramped up as losses increased as well, as did political spending (FTX spent much more on political lobbying in the months before it collapsed than previous months, IIRC). 

I actually think any analysis of FTX on this dimension without looking at FTT has a hole in it, and I might update this post, or write a follow-up one. Patio11 and Matt Levine have written about FTX and FTT and how this resulted in crazy leverage in almost the exact way I document here. 

FTX at various points, via indirect channels like Alameda, used FTT, basically it's own stock as a collateral to get more loans. The value of its own stock was largely determined by the investment it was taking on, which functionality constituted more debt. This was as Matt Levine called it "deep dark magic" of the kind I am talking about in this post. 

This full coverage: 

But sometimes market moves are catastrophic, and in particular, sometimes securities broker-dealers and crypto exchanges will have “run on the bank” risks. If everyone knows that you are in this situation — that you have a lot of Bitcoin collateral and Bitcoin prices are falling — people will expect you to have to liquidate your Bitcoin collateral, so they will expect Bitcoin prices to fall, so they will sell Bitcoin, which will cause Bitcoin prices to fall, which will cause your long-Bitcoin customers to default, which will cause you to liquidate Bitcoin at lower and lower prices, etc., until you are bankrupt.

Now let’s add one more crypto element. If you are a crypto exchange, you might issue your own crypto token. FTX issues a token called FTT. The attributes of this token are, like, it entitles you to some discounts and stuff, but the main attribute is that FTX periodically uses a portion of its profits to buy back FTT tokens. This makes FTT kind of like stock in FTX: The higher FTX’s profits are, the higher the price of FTT will be. 8  It is not actually stock in FTX — in fact FTX is a company and has stock and venture capitalists bought it, etc. — but it is a lot like stock in FTX. FTT is a bet on FTX’s future profits.

But it is also a crypto token, which means that a customer can come to you and post $100 worth of FTT as collateral and borrow $50 worth of Bitcoin, or dollars, or whatever, against that collateral, just as they would with any other token. Or something; you might set the margin requirements higher or lower, letting customers borrow 25% or 50% or 95% of the value of their FTT token collateral.

If you think of the token as “more or less stock,” and you think of a crypto exchange as a securities broker-dealer, this is completely insane. If you go to an investment bank and say “lend me $1 billion, and I will post $2 billion of your stock as collateral,” you are messing with very dark magic and they will say no. 9  The problem with this is that it is wrong-way risk. (It is also, at least sometimes, illegal.) If people start to worry about the investment bank’s financial health, its stock will go down, which means that its collateral will be less valuable, which means that its financial health will get worse, which means that its stock will go down, etc. It is a death spiral. In general it should not be possible to bankrupt an investment bank by shorting its stock. If one of the bank’s main assets is its own stock — is a leveraged bet on its own stock — then it is easy to bankrupt it by shorting its stock.

The worst case is something like:

1. 1.You have 100 Customer As who are long Bitcoin on margin: They each have 1 Bitcoin in their accounts and owe you $10,000.
2. 2.You have 100 Customer Bs who are short Bitcoin on margin: They each have $20,000 in their account and owe you 0.5 Bitcoin.
3. 3.You have loaned 50 of the Customer As’ Bitcoins to the Customer Bs, and $1 million of the Customer Bs’ dollars to the Customer As. You keep the other 50 Bitcoins and $1 million as collateral.
4. 4.Your accounts show that you owe clients 100 Bitcoins and $2 million, and that they owe you back 50 Bitcoins and $1 million, and you have 50 Bitcoins and $1 million on hand, so everything balances.
5. 5.You have one Customer C who says “hi I would like to borrow 50 Bitcoins and $1 million, I will secure that loan with 150,000 FTT, each of which is worth $20.”
6. 6.You say “sure, sounds good,” and hand over all your collateral.
7. 7.Now you have 150,000 of FTT, worth $3 million, as collateral (and no Bitcoins or dollars).
8. 8.Your accounts show that you owe clients 100 Bitcoins and $2 million and 150,000 FTT, and they owe you back 100 Bitcoins and $2 million, and you have 150,000 FTT of collateral, so everything balances.

But then if the value of FTT drops to zero, you have nothing. You have no Bitcoins to give to the customers to whom you owe Bitcoins, no dollars to give to the customers to whom you owe dollars. You just have to call up Customer C and say “hey we need all those dollars and Bitcoins back.” But Customer C will not want to give you back all those valuable dollars and Bitcoins in exchange for now-worthless FTT. Also the fact that Customer C had all that FTT in the first place is not a great sign. It is an FTT whale, and FTT is now worthless. Has it been borrowing elsewhere against FTT? Are all those debts coming due? 

Now let’s add a few more FTX-specific elements. One is that FTX is an exchange for levered traders, offering products like perpetual futures and leveraged tokens that build in margin lending. So whereas the basic model of Coinbase is “they buy Bitcoin for you and put it in an envelope,” the basic model of FTX has to be “they lend you money to buy crypto and then make use of your crypto to get the money.” In financial terms, they have to rehypothecate your collateral; you can’t expect them to just keep it in an envelope if they’re lending you the money to buy it.

The other is that FTX is closely associated with a hedge fund called Alameda Research. Sam Bankman-Fried founded Alameda to do crypto arbitrage and market-making trades, and then he founded FTX to basically have a better exchange for Alameda to trade on. Alameda has lots of FTT, and last week Coindesk reported on its balance sheet; the gist of that report was “wow its balance sheet is mostly FTT”:

The financials make concrete what industry-watchers already suspect: Alameda is big. As of June 30, the company’s assets amounted to $14.6 billion. Its single biggest asset: $3.66 billion of “unlocked FTT.” The third-largest entry on the assets side of the accounting ledger? A $2.16 billion pile of “FTT collateral.”

There are more FTX tokens among its $8 billion of liabilities: $292 million of “locked FTT.” (The liabilities are dominated by $7.4 billion of loans.)

That is not in itself a reason for a run on FTX! It might be a reason for the price of FTT to go down, if you think that Alameda has too much of it and might need to sell it.

The reason for a run on FTX is that you think that Alameda is, in my terminology, Customer C. The reason for a run on FTX is if you think that FTX loaned Alameda a bunch of customer assets and got back FTT in exchange. If that’s the case, then a crash in the price of FTT will destabilize FTX. If you’re worried about that, you should take your money out of FTX before the crash. If everyone is worried about that, they will all take their money out of FTX. But FTX doesn’t have their money; it has FTT, and a loan to Alameda. If they all take their money out, that’s a bank run.

And all of this is self-fulfilling: If you are worried about FTX’s business, then the price of FTT should go down. If the price of FTT goes down, then FTX’s business is riskier, because it has less collateral. If, say, the operator of the biggest crypto exchange gently raises one eyebrow and says “FTT, eh?” that can be enough to topple FTX. FTT goes down, leaving FTX undercapitalized, leading to customer withdrawals, leading to ruin.

Anyway it is still early and confusing but that seems to be the story of FTX.

E.g. building good tooling for alignment research doesn't require this at all. 

What do you mean, of course it does, or at least something close to it? If you don't care about it you just take the highest paying job, which will definitely not be to build good tooling for alignment research! Motivation is a necessary component for doing good work, and if you aren't motivated to do good work by my lights, then you aren't going to do good work, so good motivations are indeed necessary.

I mean, I think no, if truly there is only a relatively small fraction of people like that around, we as the moderators can just ask those people to leave. Like, it's fine if we have to ask hundreds of people to leave, the world is wide and big. If most of the internet is on board with not having this specific stipulation, then there is a viable LessWrong that doesn't have those people.

We are saying that there is an obvious conflict of interest when an author removes a highly upvoted piece of criticism. Humans being biased when presented with COIs is common sense, so connecting such author moderation with rationality is natural, not a weird rhetorical move.

Look, we've had these conversations. 

I am saying the people who are moderating the spaces have the obvious information advantage about their own preferences and about what it's actually like to engage with an interlocutor, plus the motivation advantage to actually deal with it. "It's common sense that the best decisions get made by people with skin in the game and who are most involved with the actual consequences of the relevant decision". And "it's common sense that CEOs of organizations make hiring and firing decisions for the people they work with, boards don't make good firing decisions, the same applies to forums and moderators".

This is a discussion as old as time in business and governance and whatever. Framing your position as "common sense" is indeed just a rhetorical move, and I have no problem framing the opposite position in just as much of an "obvious" fashion. Turns out, neither position obviously dominates by common sense! Smart people exist in both sides of this debate. I am not against having it again, and I have my own takes on it, but please don't try to frame this as some kind of foregone conclusion in which you have the high ground.

The rest of your comment seems to be forgetting that I'm only complaining about authors having COI when it comes to moderation, not about all moderation in general.

I was (and largely am) modeling you as being generically opposed to basically any non-spam bans or deletions on the site. Indeed, as I think we've discussed, the kind of positions that you express in this thread suggest to me that you should be more opposed to site-wide bans than author bans (since site-wide bans truly make counterveiling perspectives harder to find instead of driving them from the comment sections to top-level posts).

If you aren't against site-wide bans, I do think that's a pretty different situation! I certainly didn't feel like I was empowered to moderate more in our conversations on moderation over the last year. It seemed to me you wanted both less individual author moderation, and less admin moderation for anything that isn't spam. Indeed, I am pretty sure, though I can't find it, that you said that LW moderation really should only establish a very basic level of protection against spam and basic norms of discourse, but shouldn't do much beyond that, but I might be misremembering. 

If you do support moderation, I would be curious about you DMing me some example of users you think we should ban, or non-spam comments we should delete. My current model of you doesn't really think those exist.