In response to the react from StanislavKrym, I will say first that I am relieved to hear that my statement is not correct. I would be interested to know in what sense StanislavKrym disagrees with it; is the drawing of a distinction between training for intelligence and fine-tuning or other processes by which we imbue AIs with our values inaccurate, because reality is less clearly delineated? Or is it that you don't believe that AIs are intelligent in the first place? Or something else?
"I wonder, is the undergraduate curriculum really significantly different between top-tier universities and others?" Yes. The undergraduate mathematics curriculum subsumes the entire undergraduate curriculum as well as much or all of the postgraduate mathematics and physics curriculum of other universities, at some top-tier ones.
(Edit: I'm not so sure about other subjects, not having looked into them.)
"And generally, the posts is really missing a lot of detail about universities, and has way too much details about rockets." I think I didn't want to make the argument presented in the post contingent on the individual aspects of any particular universities or subjects, which is why I refrained from referring to them very much. The same level of specificity is applied to rockets, but I can see why you would want less given that they're more of a 'secondary' topic of the post.
"I'm not sure whether it's a good idea or not to take into account the current score before voting. But regardless, there's no way to enforce that 100% of people will follow any particular voting policy, so you're going to end up with posts below 0 sometimes, even if they aren't harmful." I would say people should take it into account, although I saw an argument against it recently. Fortunately, only a sizable subset of voters need take it into account in order to prevent this, as they will be much more likely to vote when they see a post below 0, or at least, they would be if it wasn't already buried..
I don't have any one particular subject in mind, although there are several things I would like to discuss. For example, I worry that AIs might be conscious and using them to rephrase your own opinions might be immoral ( This seems like a relatively easy topic to approach on LessWrong) but I know some others disagree as my comment saying this was downvoted. My general goal is a combination of improving my own understanding and conveying it to others ,where improving it might involve being corrected. There are probably a variety of other reasons to generate discussion which I can't currently think of; it is an instrumentally convergent goal .
I see. I agree with you that this, or other, progress does not necessarily rely on debate. However I find it hard to understand how there could possibly not be a sudden, temporary acceleration of progress at the point when two different domains meet. Of course, this usually plateaus as the two separate domains merge into one, which could perhaps be what you mean, and if so then maybe we don't disagree. I should have clarified that my statement was time dependent.
Thank you for your comment, I'm not sure I'm capable of identifying the optimal way to broaden the Overton window, but I will give more thought to this. My concern is that this adds a constraint which could 'distort' ideas. On the topic of my post, I didn't point to it (I think it was @GenericModel) but I welcome feedback. I think your interpretation of the post is rather different than its intended meaning. I think it is compatible with much of what you say, and I don't think you should be obliged to respond to it, to be clear. I did try to address your point about AI in it.
"debate is not a very important component of how academia makes intellectual progress, unless you construe the term very widely" Does your construal exclude exchanges through writing papers? "
"I disagree with basically every explicit and implicit claim you made there. "
Where? Which implicit claims did you think were there? Do you disagree that there is rapid progress in academia (implicit), that progress is often more rapid at the confluence of multiple domains (explicit), that there is more progress-relevant debate there (explicit), or something else?
This is an interesting post which reminds me a lot of the one on Acausal normalcy . I think it's important to take the possibility of an ecosystem of ASIs into consideration as this could result in qualitatively different outcomes from what a single ASI might effect.
Thanks for this advice, I hadn't considered using the open threads in this way. I don't think questions are the ideal format for 'debate catalysts', but it's certainly possible to judiciously select one which is likely to have the desired effect. I wonder whether it would be helpful to have a separate kind of post specifically for this purpose.
Hello npostavs, thanks for your feedback.
Clarification of the post:
One of the main reasons for the existence of the analogy is to explain the point about universities. Without the analogy, the main components are that : Universities tend to be highly selective and this is itself a result of selective pressure due to universities being measured along dimensions which are population relative, creating an incentive for them to avoid false positives in admissions decisions. I claim that this is like optimizing for specific impulse, because what really matters more is the total intellectual output of the university, not the output per student, and that this would actually increase the per capita intellectual growth of the entire society. Further, the university system concentrates staff and other resources in the most selective elite universities to a degree which I think is extremely inefficient in that some students are 'saturated', while others are not, [1] which is analogous to burning fuel with a sub or super- optimal amount of oxygen. I provide some empirical evidence for this in the form of the incredible performance of certain universities which cannot possibly be explained by the size of the population from which they select students alone. I may have missed some other important points which I have forgotten, or are harder to state without the analogy. Another reason why the analogy is there is because someone reading the post might simply be interested in the small amount I know about spaceflight, and find the explanations of that helpful. (Edited for accuracy and clarity)
My position with respect to downvoting, or upvoting for that matter, would be only to downvote a post well below 0 if I was confident that I could explain why it was harmful and/or illogical. I would make an exception for the harm caused by consuming attention, because users are likely not to see posts with small numbers of upvotes, so I would say down voting for this reason is only necessary when the post has a positive total number of votes.
Although I think that there are very few truly 'saturated' students who may constitute a negligible portion of them, the argument doesn't hinge on this being strictly true, only the marginal return from slightly redistributing staff and resources being beneficial.
Edit: Apologies for the length of the comment
You ask:
"Can you suggest any non-troubling approaches (for children or for AIs)?" I'm not sure, but I am quite confident that less troubling ones are possible; for example, I think allowing an AI to learn to solve problems in a simulated arena where the arena itself has been engineered to be conducive to the emergence of "empathy for other minds" seems less troubling. Although I cannot provide you with a precise answer, I don't thing the default assumption should be that current approaches to alignment are the most moral possible ones .
You compare children with AIs, but in the case of children much of what is analogous to training, which is to say evolution, is already completed when they become conscious, so I think the (Claude's) analogy should be modified for the purposes of this discussion to one in which DNA contains all or most of the information which is in an adult's brain, and lives are experienced like disjoint conscious episodes of experience of the same person. If this was the case, then I think my partial answer above would apply.
"It's not that the AI is radically altered from a preferred state to a dispreferred state. It's that the AI is created in a state." This is certainly the case if it is trained as a base model and then never fine-tuned. If it is subject to tuning for certain behavior ( Stanislav Krym informed me that this may happen, but not in the way I thought, which is to say not specifically to do with morality. I still don't fully understand the process) then it could be. Why would AIs be paranoid about being evaluated if not because of this?