I probably agree.

I'd also be interested in a comparison with chemicals like propylene glycol with respect to biohardening public spaces.

It's been a while since I read AI 2027, but the bioweapons were part of it. I felt the depiction of how AI accelerated AI research more realistic in Red Heart. AI 2027 leaned on hacking more than Red Heart too. The AI in Red Heart felt more like a difficult enemy that could in theory be beaten by the main character, rather than a superhuman god doing research beyond the comprehension of mere humans. I'm not saying AI 2027 is lacking in a technical sense. Just that to a layperson, it would trigger more skepticism than Red Heart.

Thanks for the clarifications.

I'm not sure I believe that having some shards that want corrigibility work to overcome the shards that conflict is a useful strategy if we don't know how to make any of the shards want corrigibility in the first place.

The alternate timeline definitely makes the timeline feel more realistic, so thanks for pointing that out.

I think the downsides are maybe just more visible to you than the upsides. To me, avoiding romantic relationships because I might have to provide 60% of the household's income, or because I have to deal with them being sad or upset sometimes, that seems crazy. (Barring an unreasonable partner, who should be straightforward to avoid. In particular, if you're dating within the LessWrong neurotype, this is less likely to be an issue, and even less so if you're specifically selecting for it.) Likewise, I'd never want to give up love just because I have an hour less a day to spend on other things because I want to spend time with my girlfriend. That hour is valuable to me and worth the break from work.

I think you are right that you'd end up somewhat less ambitious, not that there's a shortage of highly ambitious people who can also feel love. The pattern you pointed to about people being basically satisfied with mediocre performance in life as long as they feel companionate love, I think that's basically a pattern relegated to those who are societal losers anyway, and love is just a consolation prize. It's not like cannabis where it makes you unambitious. I'd guess this trap is most common among the working class, where most adults have no realistic shot of success anyway, so they concern themselves more with relationships and friendships.

I'm kind of saying that you're at least half-right about all your reservations, but that it would be the right decision (IMO) to make the change anyway, if possible. According to your own values, because it's not that you know what you're missing and are choosing not to pursue it according to your values. It's that you don't know what you're missing.

Btw companionate love feels pretty satisfice-able to me. Not sure if anyone else feels differently. But I'm in a steady state with my girlfriend, where I feel the desire to spend a certain percentage of my time with her, and beyond that I feel like I'm satisfied, and she feels similarly. More time together after that is nice in the same way that more snack food might still taste somewhat good even when you're not really hungry anymore.

I think you're conflating your perspective and your values. I think companionate love is not desirable from your perspective, but not necessarily from your values. Think about someone who couldn't feel joy (or pleasure or whatever). They would be saying the same things you're saying now, and they would be wrong, too. Just because a person hasn't felt joy before doesn't mean they don't value joy. I would say if a non-joy-feeling person could take a drug to make them feel joy, they wouldn't be altering their values. Their value function already included a variable for joy; that variable was just always set to zero. Their perspective on joy would change, but they already liked feeling good. Now they just have a new way of feeling good. The desire for "positive experience" is deeper than desire for specific positive feelings you can have, and probably a value you already have.

You could possibly say the same thing about regular people who haven't yet discovered the joy of injecting heroin, but I think the downsides of taking heroin are clear and outweigh the upsides (hence the number of people who regret trying heroin and the dearth of people who think heroin is just awesome). Heroin clearly destroys too much of what you already care about. Maybe you're worried the same thing might be true of companionate love?

Then let me try to sell you on companionate love: It makes me feel like I'm not alone in the world. During times when I've been lonely, I felt purposeless, like any thoughts I had, anything I wrote, anything happening in my life, it all just happened and then faded into the past. But with people I love, I feel like my life has meaning in that it has an impression on them, and their lives have an impression on me, in a way that reduces anxiety and makes life meaningful and full. It makes me feel supported in life. Obviously without companionate love you can still have relationships and practical support, but I'm talking about something more than that. Being a part of something larger than yourself. Caring about more than just yourself (not through morality or empathy or guilt, but actually caring directly). And being cared for in the same way, and feeling that.

The downsides are minimal. Some time, and some obligation to others (which is usually repaid in kind anyway). Most people who can feel companionate love are able to avoid the trap of money-hungry, emotionally-burdensome romantic partners. The people who don't are making an avoidable mistake. (Just make sure the person you love is kind and compassionate before you commit to them. And usually I think it's passionate love/anxious attachment that leads to such mistakes anyway.) I suspect almost nobody on Earth, perhaps not a single person, would give up their ability to feel companionate love just to avoid this trap. Would you give up your ability to feel happiness just so you could free up 5% of your time for working?

If I lost my ability to feel companionate love, I would consider that a loss on the order of losing my ability to feel joy. If it's not possible to repair this deficit then so be it, but if it was possible, I would think it urgent to do so. The same way if someone was born without pleasure, you would urge them to fix that, even if they didn't understand why this pleasure thing was so important.

I am definitely skeptical of the idea that we should encourage someone to sacrifice their ability to feel love on the altar of "potentially future-critical insights" they might have. The AI future is too hazy to be demanding such sacrifices at this point. Especially since fixing the oxytocin issue probably wouldn't impact his ability to do good research, unless you'd suggest others should be finding ways to lower their oxytocin to become hyperfocused in the same way.

Thanks for your detailed response. I agree that if we have enough data/compute, we could overcome the data/compute inefficiency of AI models. I suspect the AI models are so intensely data/compute inefficient that this will be very difficult though, and that's what I tried to gesture at in my post. If I could prove it, I'd have written a white paper or something instead of a blog post, but I hoped to at least share some of my thoughts on the subject.

Some specific responses:

1. Just increasing compute. I agree this is why we measure loss, but that doesn't imply that measuring loss will get us to superintelligence long-term. Also, for this: "benchmark performance, which is nonzero only for models large enough", I think you could have benchmarks that scale with the model, like novel games that start simple, and grow more complex as the model gains capability. Either manually, or implicitly as with AlphaGo Zero.
2. Higher quality data. Thanks for bringing my attention to CALM, I'll have to look into that. I don't think using a not-so-intelligent LLM to check whether the student's idea and the real idea are the same will work in the limit, for the same reason it would be hard to get a kindergartner to grade a high school math test, even if they had access to a correct version written by the teacher. (Assuming the test wasn't multiple choice, or single numerical answers or something easy to verify.)
3. Using another smaller LLM as an evaluator. I'm definitely not against all approaches that use a smaller LLM to evaluate a larger LLM, and you're right to push back here. In fact, I almost suggested one such approach in my "what might work" section. Narrow models like AlphaGo Zero do something like this to great effect. What I'm against specifically is asking smaller models to evaluate the "goodness" of an output, and trusting the smaller LLM to have good judgement about what is good. If it had to judge something specific and objective, that would possibly work. You want to trust the small model only for what it's good at (parsing sentence structure/basic meaning of outputs, for example) and not what it's bad at.
4. RLHF. RLHF works for what it does, but no amount of RLHF can overcome the problems with self-supervised learning I discussed in the post. It's still a general "reality-stuffing" model. That's all I meant.
5. Transformers and "attention". I do not take the benchmarks like solving the IMO seriously. These same AI models fail to solve kindergarten math worksheets, and fail to solve very basic problems in practice all the time. In particular, it does not seem smart to test how well a model can think by giving it problems that may require a whole lot of thinking, or may require not much, depending on what similar things happened to be in the training data, which we have no idea about. You mentioned P=NP. Solving problems is much easier if you already know how to solve similar-enough problems. We don't know what similar problems a given model does or does not know how to solve. Rendering the benchmark useless. Unless you construct a benchmark such that we know there can't have been anything meaningfully similar in the training data (e.g. novel games). (I am unsure whether to take FrontierMath Tier 4 a bit more seriously because the problems seem really hard and unlikely to be similar to anything in the training data, but ideally you'd have a benchmark that works even for less difficult problems anyway.) As for your comment about online learning, I don't think solving any particular task should require a model to totally reorganize its weights across the entire model. Updating only a little part of weights should be fine. An analogy to humans should show that much. I agree though that having to hold onto fine-tuned partial models for users, even briefly, is more expensive than what we're doing now, but the capabilities gains may eventually be worth it if non-online-learning models do plateau.

It was subtle. I could just sometimes sense the prickly, digital nature of my sense of touch if I moved it smoothly along e.g. a blanket.

I managed to find one of the Sam Harris clips where he talks about this

...even in the immediate aftermath of an atrocity like this. This will come as no surprise. They will tell you that this has nothing to do with Islam. It has nothing to do with heartfelt religious convictions. No, it has everything to do with capitalism and the oppression of minorities, and the racism of white people in Europe, and the racism of cartoonists at a magazine like Charlie Hebdo. That is the cause of this behaviour. That's what causes someone to grab an AK-47 and murder 12 cartoonists and then scream Allahu Akbar in the streets. It is a completely insane analysis.

I get the sense that people still don't understand what we're dealing with here. Have you seen any of these interviews with captured Isis fighters? Religion is the whole story. They are totally fixated on getting into paradise. In fact, the Kurds have put female soldiers into the field, and this terrifies members of Isis because they believe that they won't go to paradise if they get killed by a woman. They literally run away from these female soldiers.