All of Aiyen's Comments + Replies

Even if we assume that's true (it seems reasonable, though less capable AIs might blunder on this point, whether by failing to understand the need to act nice, failing to understand how to act nice or believing themselves to be in a winning position before they actually are), what does an AI need to do to get in a winning position?  And how easy is it to make those moves without them being seen as hostile?  

An unfriendly AI can sit on its server saying "I love mankind and want to serve it" all day long, and unless we have solid neural net interpr... (read more)

  It does, and a superintelligence will understand those facts better than we do.

What odds are you willing to give taking friction into account?

My personal confidence of "no aliens" is so high it rounds to 100%. Placing a bet is basically just a loan with a weird "if aliens are real I don't get paid back" tacked on. The real question then is "at what rate am I willing to lend $X0,000 for 10 years to a stranger? If you can guarantee that I'll beat the stock market by 5% (conditional on no aliens) then I'm good to go, but the "guarantee" is very important. It needs to take into account things like bankruptcy on your end. I don't think our spread is wide enough to make that feasible. It'd take a lot of paperwork. This is why we need real, formal, legal prediction markets, with derivatives. They would solve all of these problems.

How much are you willing to bet? I’ll take you up on that up to fairly high stakes.

I agree that alien visits are fairly unlikely, but not 99% unlikely.

I'm willing to bet five figures, in theory, but there's a ton of factors that need to be accounted for like capital tie-up, counterparty risk, the value of my time, etc. So if your odds aren't lower than 90%, then it's probably not even worthwhile to bet. Too much friction.
  1. This seems untrue. For one thing, high-powered AI is in a lot more hands than nuclear weapons. For another, nukes are well-understood, and in a sense boring. They won’t provoke as strong of a “burn it down for the lolz” response as AI will.

  2. Even experts like Yann LeCun often do not merely not understand the danger, they actively rationalize against understanding it. The risks are simply not understood or accepted outside of a very small number of people.

  3. Remember the backlash around Sydney/Bing? Didn’t stop her creation. Also, the idea that gov

... (read more)
1Michael Simkin10mo
1. The AI in hands of many humans is safe (relatively to its capabilities), the AI that might be unsafe needs to be developed independently.  2. LeCun sees the danger, he claims rightfully that the danger can be avoided with proper training procedures.  3. Sydney was stopped because it was becoming evil and before we knew how to add a reinforcement layer. Bing is in active development, and is not on the market because they are currently can't manage to make it safe enough. Governments install regulations to all major industries, cars, planes, weapons etc. etc. it's good enough for the claim that just like cars are regulated today, future AI based robots, and therefor the AIs themselves will be regulated as well. 4. Answer me this: can an AI play the best chess moves? If you agree with this claim, that no matter how "interesting" some moves seems, how original or sophisticated, it will not be made by a chess engine which is trained to maximize his winning chances. If this sounds trivial to you - the goal of engines trained with RLHF is to maximize their approval by humans. They are incapable to develop any other agenda alongside this designed goal. Unlike humans that by nature have several psychological mechanisms, like self interest, survival instinct etc. those machines don't have those. Blaming machines of Goodharting, it's just classical anthropomorphism, they don't have any other goal than what they were trained for with RLHF. No one actually jailbreak chatGPT, this is a cheap gimmick, you can't jailbreak it, and ask to tell you how to make a bomb - it won't. I described what jailbreaking is in another comment, it's far from what you imagine - but yes sometimes people still succeed in some level of wanting to harm humans (in an imaginary story when people ask it to tell them this story). I think for now I would like to hear such stories, but I wouldn't want robots walking around not knowing if they live in reality or simulation, open to the possibility to ac
Answer by AiyenMar 05, 202310

Exercise and stimulants tend to heighten positive emotions. They don’t generally heighten negative ones, but that’s probably all to the good, right? Increased social interaction, both in terms of time and in terms of emotional closeness, tends to heighten both positive and negative emotions.

Strongly upvoted. This is a very good point.

150,000 people die every day.  That's not a small price for any delays to AGI development.  Now, we need to do this right:  AGI without alignment just kills everyone; it doesn't solve anything.  But the faster we get aligned AI, the better.  And trying to slow down capabilities research without much thought into the endgame seems remarkably callous.  

Eliezer has mentioned the idea of trying to invent a new paradigm for AI, outside of the conventional neural net/backpropagation model.  The context was more "what would you ... (read more)

2Gerald Monroe1y
Another general way to look at it is think about what a policy IS. A policy is the set of rules any AI system uses to derive the input from the output. It's how a human or humanoid robot walks, or talks, or any intelligent act. (Since there are rules that update the rules) Well any non trivial task you realize the policy HAS to account for thousands of variables, including intermediates generated during the policy calculation. It trivially, for any "competitive" policy that does complex things, can exceed the complexity that a human being can grok. So no matter the method you use to generate a policy it will exceed your ability to review it, and COULD contain "if condition do bad thing" in it.

This is excellently written, and the sort of thing a lot of people will benefit from hearing. Well done Zvi.

Well said! Though it raises a question: how can we tell when such defenses are serving truth vs defending an error?

As for an easier word for “memetic immune system”, Lewis might well have called it Convention, as convention is when we disregard memes outside our normal mileu. Can’t say for Chesterton or Aquinas; I’m fairly familiar with Lewis, but much less so with the others apart from some of their memes like Chesterton’s Fence.

Good analogy, but I think it breaks down. The politician’s syllogism, and the resulting policies, are bad because they tend to make the world worse. I would say that Richard’s comment is an improvement, even if you think it might be a suboptimal one, and that pushing back against improvements tends to result in fewer improvements. Don’t let the perfect be the enemy of the good is a saying for very good reason.

The syllogism here is more like:

  1. Something beneficial ought to be done

  2. This is beneficial.

  3. Therefore I probably ought not to oppose this, though if I see a better option I’ll do that instead of doubling down on this.

It could be that Richard's comment is actually good. I still think that the argument I quoted fails to establish that, for the same reason the politician's syllogism doesn't work.

How functional can our community be without pushing back against people like Ziz? Richard’s comment seems to be a way of doing so, and thus potentially useful. It’s fine if you disagree with him, but while I agree the comment was flag-planting, some degree of flag-planting is likely necessary for a healthy discussion. Consider the way well kept gardens die by pacifism (can’t link on my phone, but if you’re not familiar with it there’s an excellent Yudkowsky post of that name that seems relevant). Zizianism is something worth planting a few flags to stop.

How functional can our community be without pushing back against people like Ziz? Richard’s comment seems to be a way of doing so, and thus potentially useful.

This is basically the politician's syllogism:

  1. We must do something.
  2. This is something.
  3. Therefore, we must do this.

In general, the politician's syllogism fails because not only must we do something, but we must do something that works and doesn't cause side effects that are worse than its benefits and doesn't have too high opportunity costs etc. In this case, it's valuable for people to "push ba... (read more)

This is the sort of work we need to be doing to understand neural nets. Excellent job!

The wanting vs liking distinction seems relevant here.  Politics can be truly fun, especially when you're discussing it with someone who's clearly presenting their views in good faith, and when you can both learn something from the interaction.  However, it's easy for the wanting to stay strong long after the liking has completely disappeared.  

I wonder if that's a common trait of most or all addictive things, or at least of "non-physical" addictions (things where you don't suffer withdrawals, yet still may find yourself spending more time o... (read more)

3Adam Zerner1y
Wow, I really like that wanting vs liking distinction! Great point! It's something that I think about a lot myself but didn't really see the relation here. I do agree that it is relevant though for the reasons you described.

It is-for a certain type of unstable person. Ziz would likely have come up with different crazy ideas without Less Wrong. Compare Deepak Chopra on quantum mechanics: he pushes all manner of “quantum” bullshit, yet you can hardly blame physics for this, and if physics weren’t known, Chopra would almost certainly just be pushing a different flavor of insanity.

More like "enjoy the dive!"  

Agreed, I was just being flippant (and quoting Oglaf). In fact I wholeheartedly endorse this post. Your version is much better. 

Combating bad regulation isn’t a solution, but a description of a property you’d want a solution to have.

Or more specifically, while you could perhaps lobby against particular destructive policies, this article is pushing for “helping [government actors] take good actions”, but given the track record of government actions, it would make far more sense to help them take no action. Pushing for political action without a plan to steer that action in a positive direction is much like pushing for AI capabilities without a plan for alignment… which we both agre... (read more)

How do you propose nudging regulation to be better without nudging for more regulation?

0Zach Stein-Perlman1y
Combating bad regulation would be the obvious way. In seriousness, I haven’t focused on interventions to improve regulation yet— I just noticed a thing about public opinion and wrote it. (And again, some possible regulations would be good.)

Regulation in most other areas has been counterproductive.  In AI, it will likely be even more so:  there's at least some understanding of e.g. medicine by both the public and our rulers, but most people have no idea about the details of alignment.  

This could easily backfire in countless ways.  It could drive researchers out of the field, it could mandate "alignment" procedures that don't actually help and get in the way of finding procedures that do, it could create requirements for AIs to say what is socially desirable instead of wha... (read more)

1Zach Stein-Perlman1y
I didn't call for regulation. Some possible regulations would be good and some would be bad. I do endorse trying to nudge regulation to be better than the default.

There's potentially an aspect of this dynamic that you're missing.  To think an opponent is making a mistake is not the same thing as them not being your opponent (as you yourself point out quite rightly, people with the same terminal goals can still come into conflict around differences in beliefs about the best instrumental ways to attain them), and to think someone is the enemy in a conflict is not the same thing as thinking that they aren't making mistakes.  

To the extent that Mistake/Conflict Theory is pointing at a real and useful dichotomy... (read more)

I agree (as I'd already said) that there isn't a nice dichotomy where some people see their opponents as beings of pure evil who do what they do solely because they are bad, and others see them as simply mistaken and therefore not in any way opposed. I am not convinced that this in any way means that in a dissidents/quislings situation you will get the dichotomy Zack claims, and once again I point to the various examples I've given; I think that in all of them (and also the two more suggested by Viliam) the quislings will typically have just as conflict-y an attitude as the dissenters. (I think the key distinction between a mistake theorist and a conflict theorist is: the mistake theorist thinks that it will be helpful for their purposes to address the other side with evidence and arguments, and try to arrive at mutual understanding; the conflict theorist thinks that won't do any good, or cares more about playing to the gallery, or whatever.) For the avoidance of doubt, I don't disagree that in some cases the quislings[1] will think that the dissenters[1] are (evil and hateful because they are) honestly mistaken. But I also think that in some cases the dissenters will think that the quislings are (evil and hateful because they are) honestly mistaken. The terminology may be unhelpful here; cases in which the word "quisling" seems appropriate will tend to be those where we think of the people in question as doing things they know are bad out of self-interest. But e.g. I bet plenty of those neoreactionaries and revolutionary communists think the advocates of liberal democracy are mostly honestly mistaken. [1] It's probably obvious but I'll say this explicitly: I do not intend either "quislings" or "dissenters" to carry any particular presumption of rightness/wrongness/goodness/badness. "Quisling" means "person functioning as some sort of spokesperson for whatever ideas are held by the people and institutions with power" and "dissenter" means "person vigorously disa

That’s a documentary about factory farming, yes? What people do to lower animals doesn’t necessarily reflect what they’ll do to their own species. Most people here want to exterminate mosquitoes to fight diseases like malaria. Most people here do not want to exterminate human beings.

"Men care for what they, themselves, expect to suffer or gain; and so long as they do not expect it to redound upon themselves, their cruelty and carelessness is without limit."-Quirinus Quirrell

This seems likely, but what is your evidence for it?  

9andrew sauer1y
For one, the documentary Dominion seems to bear this out pretty well. This is certainly an "ideal" situation where cruelty and carelessness will never rebound upon the people carrying it out.

This is blatantly wrong.  Restricting the lives of other people just to gain a little more cachet for you and your fellow urbanites.  Clear defection, clear evil. 

What is the specific difference between “regurgitated” information and the information a smart human can produce?

The human mind appears to use predictive processing to navigate the world, i.e. it has a neural net that predicts what it will see next, compares this to what it actually sees, and makes adjustments. This is enough for human intelligence because it is human intelligence.

What, specifically, is the difference between that and how a modern neural net functions?

If we saw a human artist paint like modern AI, we’d say they were tremendously talented.... (read more) 

I would like to leave this here as evidence that the model stated above is not merely right on track, but arguably too conservative.  I was expecting this level of performance in mid 2023, not to see it in January with a system from last year!  

Have you tried this before? It sounds potentially helpful, but there’s nothing about what you’ve achieved with this method, only why it might work.

This is true, but it doesn’t answer the question of why not to simply use nuclear blackmail on such states. And the answer to that is that the US wants to limit the destruction of war. Nuclear blackmail is great, right up until someone calls your bluff. But then it helps to have conventional forces if you do not wish to have massive losses to local civilians, local infrastructure, and one’s own prestige.

The main reason USA (and other nuclear powers) don't use nuclear blackmail is that it would end no-nuclear-proliferation regime. "Every state that can make nukes has them" is the natural word state, keeping non-proliferation requires effort.
Answer by AiyenDec 28, 2022126

"There are many animals which have what are called dominance contests. They rush at each other with horns - trying to knock each other down, not gore each other. They fight with their paws - with claws sheathed. But why with their claws sheathed? Surely, if they used their claws, they would stand a better chance of winning? But then their enemy might unsheathe their claws as well, and instead of resolving the dominance contest with a winner and a loser, both of them might be severely hurt." -Professor Quirell

Or to be more explicit, anything less than total... (read more)

Citation very much needed. What, specifically, do you disagree with?

Do you believe that the human mind is magical, such that no computer could ever replicate intelligence? (And never mind the ability it has shown already from chemistry to StarCraft…)

Do you believe that intelligence cannot create better tools than already exist, such that an AI couldn’t use engineering to meaningful effect? How about persuasion?

Do you believe that automation taking over the economy wouldn’t be a big deal? How about taking over genetics research, which is often bottlenec... (read more)

7the gears to ascension1y
I think that the vibes here have been unhealthy in ways that made model-challenging evidence seeking harder to think of; for example, yudkowsky's intense anxiety, dying with dignity needing to be said and also that shock inducing name, etc. Your reaction seems to me to miss the point that I would agree with CellBioGiy about, though I do also think you're right in every implied rebuttal your questions point to; it's just that trying to solve safety requires willingness to question deep assumptions and yet retain stability in the face of doing so, and retaining stability is hard and often failed at when taking huge magnitude ratios seriously. Ziz seems to me to have been the most intense example of a general failure pattern in how this intercommunity directs a person's intention, and thinking about that carefully seems important to me. It's also something where real progress has been made and I hardly think the critique warrants writing off the site as a whole or anything like that. I do think calling the whole thing a spiraling cult takes it too far; it's got elevated levels of cult disease and features that can't be removed which make it pattern match aesthetically to cult adjacency in ways that may not always be fundamental - real crazy things are coming - but I think that there are in fact concerning thought stopping patterns in the interpersonal anxiety patterns, patterns of who trusts who without ongoing verification, etc.

Fair enough, but it is equally incomplete to pretend that that’s an argument against the possibility of singularity-grade technology emerging in the foreseeable future.

By analogy, there have been many people who had crazy beliefs about radioactivity: doctors who prescribed radium as medicine, seemingly on the grounds that it was cool, and anything cool has to be good for you right? (A similar mentality led some of the ancient Chinese to drink mercury.) Atomic maximalists, who thought that anything and everything would get better with a reactor strapped ... (read more)


While true, that’s not actually relevant here. While LW does not have perfect agreement on exactly how morality works, we can generally agree that preventing vaccine waste is a good idea (at least insofar as we expect the vaccine to be net-beneficial, and any debates there are largely empirical disagreements, not moral ones). Nearly all consequentialists will agree (more people protected), as well as deontologists (it’s generally desirable to save lives, and there’s no rule against doing so by utilizing vaccines that would otherwise end up in the trash) ... (read more)

2M. Y. Zuo1y
The opportunity cost of the resources and time they used up was not zero nor was that the only possible thing they could have done.  It only seems like the definite correct thing to do because no alternative possibilities were considered in comparison. Even within the context of saving lives during the vaccine rollout, there may have been alternative course of action that led to even more saved. In the broader context of improving the human condition generally, the common rejoinder on LW, at least in the past, could have included things such as 'malaria nets', 'QUALYs', etc... This line of reasoning has been discussed so many times on LW it's surprising you haven't come across it

Strongly upvoted for clarification and much greater plausibility given that clarification.  

"Back then it was called Czechoslovakia. I am puzzled about the disagreement votes, given that I have hedged my statement as "try to teach you, even if not very efficiently". Not sure how people do things on the other side of the planet, but I imagine that there are these things called textbooks, which are full of information, and they at least make you read them. I am not saying that the information is especially useful, or especially well explained; just that... (read more)

Well, what you describe is much worse than I imagined... and I still have a problem imagining that this is a typical American school experience. No offense, but I would need more people to confirm this, before I update on American education in general. I have often heard that American schools are "teaching to the test". Would you agree with that statement? If yes, how does that square with... not even reading the textbooks, and adding negative numbers in seventh grade? Were the seventh grade tests about adding negative numbers? On the other hand, I have also heard that as a consequence of "No Child Left Behind", teachers are focusing on the students that perform worst. So, maybe the least performing student in your seventh grade had a problem with adding negative numbers? That would kinda explain. Or do the both things combined together mean that American teachers are "teaching the worst performing student to the test"? Sad, but possible, given the incentives. ("No Child Left Behind" was replaced in 2015 by a new law, hopefully better.) I admit that if I never attended school, and spent maybe 1/3 of the extra free time reading books (sounds likely, I have read tons of books as a child), I probably would have grabbed a few chemistry books, too, just out of curiosity. So maybe the benefit of the school was smaller than I imagine. I still give the school credit for curating the information: filtering out knowledge from bullshit, and ordering the knowledge from simple to more complex. In the alternative reality where I learned chemistry from library books, who knows, maybe a book written by a convincing homeopath would get to my hands first, and I would waste lots of time studying bullshit. The benefits of chemistry for my life would be: (1) It increases the general protection against bullshit. Like, when I heard about homeopathy, I remembered what I learned at school, and I noticed that this is not how atoms and molecules actually work. Water is a liquid, the mol

"Schools at least try to teach you."  

I am curious where you went to school.  That was not my experience, and I was in an unusually good school district by American standards.  Some of my friends had noticeably worse experiences than I did.  Are you conflating the nominal purpose of a school with its real-world actions?  Alternatively, did you go through a good enough school system that it might be worth replacing a great many existing "educational" systems with yours as a stopgap along the way to school abolition?  

"Jobs typi... (read more)

Back then it was called Czechoslovakia. I am puzzled about the disagreement votes, given that I have hedged my statement as "try to teach you, even if not very efficiently". Not sure how people do things on the other side of the planet, but I imagine that there are these things called textbooks, which are full of information, and they at least make you read them. I am not saying that the information is especially useful, or especially well explained; just that it is there, and the school exposes you to it. There were subjects that I hated, mostly history. That one was taught literally as a list of facts that I considered utterly irrelevant -- I couldn't care less about what year exactly which king was born, or what year exactly was a battle that happened centuries ago. There were subjects that I would have learned on my own anyway, especially math and computer science. A subject where the school provided the most added value for me, was probably chemistry. Explained sufficiently well that some information stuck in my brain, and yet not something I would have studied on my own. Okay then. Correct. Perhaps this is another case of what I call "America succeeds to go to all extremes simultaneously". Excellent universities, dystopian elementary and high schools. In Eastern Europe things are more... mediocre, at both extremes. The universities are meh, but the elementary schools are kinda okay, mostly. Or maybe one of us generalizes too much from personal experience. It would be interesting to make a survey (not limited to the rationalist community). I was seriously considering the possibility of homeschooling my children, but it turned out that my daughter enjoys school (she is currently in 2nd grade), so I am like: "well, if there is no problem, I do not need to solve it". Of course, enjoying and learning new things are not the same; she probably likes the fact that she is one of the best in the classroom. Then she does Khan Academy and Duolingo at home. That doesn

How is this different from adults having jobs?  

To be clear, there are plenty of good reasons why one might not want children to work.  You might want them to be able to enjoy childhood without the burden of a job, you might want them to focus on learning to be more productive later.  But "the people paying them are motivated by profit" is equally true of adult jobs. 

The question was how this is different from school. Adults having jobs is also different from school. In addition, the part about parents having control over children and children's money doesn't apply to parents of adults with jobs.

"Oh right, the whole world doesn't have education as a right."

Are you trying to argue from existing law to moral or practical value?  That would be easier if the whole world hadn't had slavery and monarchy until fairly recently. 

"That both destroy magic doesn't mean the destruction is it to the same degree." 

That's a good point.  But jobs ideally produce value.  School often doesn't, and "learning" in a toxic setting specifically makes it harder to learn later.  That's a harm specific to school; most jobs do not have it.

"And s... (read more)

My tone is bad and inappropriate especially in this contex. What I actually mean or should have meant is that "parents lose money" is not really descriptive of my local reality and I have trouble taking on that perspective. Trying to imagine a counterfactual what would have needed to be different to not have universal education starts to baffle me a little. My brains come up with questions like "Do people in this world have to pay the police if they call them to protect from gunmen?" which are more obviously out of touch what I know to be the case. The money loss is a facet of some corners of reality. I am familiar with the organization where teachers are first accountable to society or state rather than accountable through parents. So "What are we paying you for?" has two sides to it that I am extremely unfamiliar with. What I am used to is that the public option is mostly appropriate, so children and parents are not constantly trying to escape it. Keeping "study duty" firm has for me the most important role that a parent while having large custodial rights may not fail to educate their kid. Alternative venues are fine but they can't be shambles, they must be worthy of the dignity of the progeny of civilization (so need to pass goverment checks). So if private schools are a large part of the equation why are not parents using the customisability if they are going the hard route anyway? If students are suffering why are the parents not advocating for their kids needs? Is it because the solutions exist but are paywalled and some that need them can not get them? How come the public options gets a pass for maleducating a significant stream of citizens? Is this some kind of thing where the most prestigious places are prestigious because they are harsh (and failures are because of students and not schools) and thus misery is a sign of status? The word "value" has so different meaning in job context and school context that I am not confident on which idea this expresses

That ignores systematic problems with schooling, which even good schools will tend to suffer from:  

Teaching by class risks both losing the kids at the bottom and boring the kids at the top, whereas individual study doesn't have this problem.  

Teaching by lecture is much slower than learning by reading.  Yes, some students benefit from audio learning or need to do a thing themselves to grasp it, but those capable of learning from reading have massive amounts of time wasted, as potentially do the kinesthetic types who should really be taking ... (read more)

How is this any different from school, except that you could get paid rather than your parents losing money to pay the teachers?  There are many valid arguments against child labor (though also many valid arguments that the child should be allowed to decide for themselves), but nearly all of them apply to schooling as well.  School eliminates the time of childhood magic, actively makes it harder to be curious (many jobs would not have this effect) and you don't even get paid.  

On the "exploration - exploitation" scale, schools at least try to teach you, even if not very efficiently. Jobs typically do not try at all... and when they try, it is usually very narrow, the thing you will immediately need for your work. ("But you learn by doing", yes, but only a very narrow selection of things. If you need to learn too much, it is typically more effective for the company to hire someone else for that job.) In school there is more slack; you can spend a lot of time daydreaming, you finish earlier, you have summer holidays. At work you get responsibilities, unrealistic deadlines, for many people it is difficult to stop thinking about their job when the shift ends. So, in some sense, both school and work suck in similar ways, but school sucks relatively less. Doesn't pay you, though. But if you consider the market value of what a small child would produce, it is peanuts. For more precise discussion, we would need to be more specific. Are we talking about jobs for 6 years olds, 10 years olds, 14 years olds? Full time or part time? What is the consequence of quitting the job or getting fired? The "childhood magic" from my perspective means, importantly, not being responsible for paying the mortgage or making enough money to buy food. As long as that is your parents' responsibility, and you are allowed to completely ignore this, you are mentally a child. When the metaphorical gun is put to your temple, adulthood begins.
If "being paid" means not only that children get money, but also that the people paying them are motivated by profit, that creates bad incentives on the part of those people. If children are paid but parents still have control over the children and the money, that creates bad incentives for the parents.
Oh right, the whole world doesn't have education as a right. It does apply also to start of school. It is about developmentally appropriate environments. Schools are supposed to be where that can be a high objective. Keeping up skill development in work is rather hit and miss and can be quite narrow for profitability increasement. That both destroy magic doesn't mean the destruction is it to the same degree. And school has its own magic. Jobs tend to have way less magic of their own.

I don’t know how common loss of attention span is, but certainly reduced interest in learning occurs extremely often.

Also, potential evidence that more damage occurs than is commonly recognized: in the modern world, we generally accept that one needs to be in one’s late teens or even early twenties to handle adult life. Yet for most of human history, people took on adult responsibilities around puberty. Part of the difference may be the world becoming more complex. But how much of it is the result of locking people up in environments with very little social or intellectual stimulation until they’re 18?

The world looks exactly like one would expect it to if school stunted intellectual and emotional maturity.

I would think that it's valid, but a smaller effect than getting taught a bundle of random things in a gratuitously unpleasant way resulting in those who have been taught in school having a deep-seated fear of learning, not to mention other forms of damage.  Prior to going to school, I had an excellent attention span, even by adult standards.  After graduating high school, it took two years before I could concentrate on anything, and I still suffer from brain fog.  

Hm not sure such damage commonly happens.
Answer by AiyenDec 03, 20222611

Should society eliminate schools?  

That depends on what would replace them.  One could imagine a scenario in which schools were eliminated, no other form of learning filled the gap, and mankind ended up worse off as a result.  However, schooling in its present form seems net-negative relative to most realistic alternatives.  Much of this will focus on the US, as that is the school system I'm most familiar with, but many of the lessons should transfer. 

Much of the material covered has no conceivable use except as a wasteful signal. ... (read more)

How does society decide what subjects get taught in school?
What would you think of the argument that getting taught a bundle of random things practices learning, so that those who have been taught in school are better able to learn other things afterwards?

Is that true?  Isn't at least one clear difference that it's difficult to stop engaging in a bias, but heuristics are easier to set aside?  For example, if I think jobs in a particular field are difficult to come by, that's a heuristic, and if I have reason to believe otherwise (perhaps I know a particular hiring agent and know that they'll give me a fair interview), I'll discard it temporarily.  On the other hand, if I have a bias that a field is hard to break into, maybe I'll rationalize that even with my contact giving me a fair hearing it can't work.  It's not impossible to decide to act against a bias, but it's harder not to overcorrect.  

Where does your bias come from? 

He cites the observation that socialized firms have not taken over the economy.  That's clearly true and clearly relevant.  Your response was that you'd already explained why socialized firms might not take over even if they were productive.  What were those reasons again?  Reviewing your post, it looks like it might be the difficulty of gaining investment and brain drain from the most productive workers leaving, but both of those reasons would be strong arguments against socialization.  Rose Wrist's ideas for gaining investment an... (read more)

-1B Jacobs1y
I cited controlled experiments, you counter with an observation that I have already responded to in both the post and the comments:   You are not engaging with the evidence I cited.

The specific handwave I'm referring to is Amartya Sen's. 

"In the case of the free rider hypothesis, these 'rational fools' act based on such a narrow conception of self-interest that they don't take into account the obviously damaging long-term consequences of their behavior, both to the firm and ultimately to themselves. Normal, reasonable people - who are different to rational economic man - are usually happy to put efforts into a collective endeavor that will deliver benefits for them in the long run, even if that means foregoing some short-term ga... (read more)

3B Jacobs1y
I cite four different studies that show that the theory doesn't match the observations, Lao Mein doesn't cite anything. This is the most extreme version of being a selective skeptic.

Surely the good or bad effects of socialism are a function of policy?  Whether or not a policy arises democratically and/or revolutionarily does not change the policy itself.  This is a striking non-sequitur.  

The Scandinavian countries are indeed pretty good places to live.  This likely has nothing to do whatsoever with democracy per se, but with the fact that the Scandinavian model does not regulate to anything resembling more strongly socialist nations, despite the fact that they famously have a large welfare system.  There is n... (read more)

(Preliminary note: I see you've had a lot of downvotes in this thread; none of them is from me.) I agree that whether a policy is good or bad doesn't depend on how it arises, but what other policies come along with it may do. For instance, so far as I can tell socialism as such doesn't need to involve much in the way of totalitarianism, but governments brought in by revolutions tend to be totalitarianism whether they are left or right or something else. At least some of the harms of e.g. communism in the USSR seem to me to have been consequences of totalitarianism more than of economic policies as such. In any case -- my apologies if I wasn't clear enough about this -- the democracy-versus-revolution thing was not my main point; my main point was that there is a big difference between (say) Soviet communism (a way of running a whole country) and workers' cooperatives (a way of running a company), and this difference seems highly relevant to the question of whether the disastrousness of the USSR tells us anything about the likely consequences of organizing more companies as workers' cooperatives. A nation isn't really much like a business, despite occasional rhetoric along those lines from politicians when the policies they prefer for other reasons happen to have the shape of "run the country more like a business". And a workers' cooperative isn't much like a communist or socialist country. If you think that running a company as a cooperative makes it more likely to fail in ways parallel to ways in which communist countries commonly fail, then I think you should show your working: explain how the relevant parallels work. (Including, in particular, explaining why you think such a company is more like Venezuela or the USSR than it is like, say, Sweden.) (Why did I mention the democracy-versus-revolution thing at all? Because it seems to me that the most plausible ways of ending up with more workers' cooperatives are more democracy-like than revolution-like, and e.g
Scandinavian people think that socialism is a functional gear in having the Nordic model work. However the understandings are so different that the word "socialism" means very different things across the pond. Too little socialism and you have obvious downsides like not having universal healthcare and much distrust among population. When a country has big tradition of tempering market forces the knowledge tends to be way more practical than "boogieman" understanding. Even if some party doesn't want to call such balancing acts by particular names they have been around and are not "new".
3B Jacobs1y
I’m not handwaving anything I wrote a whole section about how experiments contradict this and what could explain this: “Experiments have shown that people randomly allocated to do tasks in groups where they can elect their leaders and/or choose their pay structures are more productive than those who are led by an unelected manager who makes pay choices for them.[20] One study looked at real firms with high levels of worker ownership of shares in the company and found that workers are keener to monitor others, making them more productive than those with low or no ownership of shares and directly contradicting the free rider hypothesis.[21] It turns out there are potential benefits to giving workers control and a stake in the running of the organization they work for. This allows workers to play a key role in decision making and reorient the goals of the organization.[22] One explanation for this phenomenon is that of "localized knowledge". According to economist Friedrich Hayek, top-down organizers have difficulty harnessing and coordinating around local knowledge, and the policies they write that are the same across a wide range of circumstances don't account for the "particular circumstances of time and place".[23] (For examples of this, read Seeing Like a State by political scientist James Scott) Those who make the top-down policies in a traditional company are different to those who have to follow them. In addition, those who manage the company are most often different to those who own the company. These groups have different incentives and accumulate different knowledge. This means that co-ops have two main advantages: Workers can harness their collective knowledge to make running the firm more effective. Workers can use their voting power to ensure the organization is more aligned with their values. Interestingly enough, I have yet to come across a co-op that uses the state of the art of social choice theory, so they could potentially get a lot lot better.“

Hence the charitable reading that the OP might be calling for a different version of socialism that might conceivably be beneficial. My point isn’t that there’s zero chance that he’s right; my point is that there’s no way to say “hey, let’s do this thing that’s superficially similar to catastrophic policies” without it either not conveying useful information, or that useful information requiring a long political debate to hash out. And that’s not appropriate for the “Politics is the mind-killer, let’s improve our rationality on easier cases” forum. I’d welcome the post and subsequent debate on e.g. a Scott Alexander forum or comment section. But this isn’t the place for it.

Aren't these obviously, non-subtly different from the stuff that has consistently led to catastrophe? Or am I missing some similarity? (I have to admit that I'm somewhat weak on the relevant history, so it's possible I'm missing some obvious deep similarity.)
As tailcalled says, there's really not much overlap between what this post is advocating and what led to (e.g.) millions of deaths under Stalin. This post is about a way a business can voluntarily choose to organize itself that might lead to better outcomes. The thing that has led to a lot of catastrophes is a way a nation can organize itself which, in practice, has only ever happened as a result of violent revolutions. (Other sorts of socialism have arisen democratically and these have not led to catastrophe; e.g., the Scandinavian countries are pretty good places to live.)
Aren't socialist co-ops a totally different kind of socialism from the stuff that has "consistently led to catastrophe"?

On the one hand, that's literally true.  On the other, I feel like the connotations are dangerous.  Existential risk is one of the worst possible things, and nearly anything is better than slightly increasing it.  However, we should be careful that that mindset doesn't lead us into Pascal's Muggings and/or burnout.  We certainly aren't likely to be able to fight existential risk if it drives us insane!  

I strongly suspect that it's not self-sacrificing researchers who will solve alignment and bring us safely through the current crisis, but ones who are able to address the situation calmly and without freaking out, even though freaking out seems potentially justified. 

Wouldn't it be relevant in that someone could recognize unproductive, toxic dynamics in their concerns about AI risk as per your point (if I understand you correctly), decide to process trauma first and then get stuck in the same sorts of traps?  While "I'm traumatized and need to fix it before I can do anything" may not sound as flashy as "My light cone is in danger from unaligned, high-powered AI and I need to fix that before I can do anything", it's just as capable of paralyzing a person, and I speak both from my own past mistakes and from those of multiple friends. 

Of course that's possible. I didn't mean to dismiss that part.

But… well, as I just wrote to Richard_Ngo:

If you just go around healing traumas willy-nilly, then you might not ever see through any particular illusion like this one if it's running in you.

Kind of like, generically working on trauma processing in general might or might not help an alcoholic quit drinking. There's some reason for hope, but it's possible to get lost in loops of navel-gazing, especially if they never ever even admit to themselves that they have a problem.

But if it's targeted, the

... (read more)
Load More