Thanks. (I have read this with interest, but don't have much of an overall reply.) A couple comments:

Humanity should count as actual AGI, but it remains slow.

It's a weird middle ground, where it is fooming but pretty slowly. (I mean, as a human, I would think that, i.e. I would experience time on a scale that's somewhat faster than humanity's foom.) And it kinda only half has the A in AGI. If it fully had the A (or I guess, ISGI, in silico general intelligence) then it would foom much faster.

Similarly, I think LLMs that do prosaic RSI of automatically building the next model (crucially including formulation of new RL tasks/environments/graders) will be actual AGI in the sense that they can (on their own, without humanity's input) eventually generate and accumulate as much crystallized intelligence as humanity would.

Maybe kinda. But my guess is that this would be kind of like calling [hominid evolution by natural selection on genetic variation] an AGI, or calling [the ecosystem of the Earth throughout all of time] an AGI. I mean it would probably be extremely faster & scarier than evolution, but still.

Pretraining very sample inefficiently reconstructs cognitive skills that left evidence about their nature in the text.

It reconstructs a lot of them to a significant extent, but very much misses a lot of them to a significant extent, I think.

But this is very likely enough for a base model of 2031 to be sufficiently teachable (using RL tasks) to end up learning how to formulate new RL tasks/environments/graders on its own, starting the slow-learning prosaic RSI process of building the next model automatically.

This seems plausible-ish. I would weakly expect a lot of plateaus, e.g. due to highly correlated taste, but not strongly and I haven't thought about it much. (Maybe you're pricing that in to "slow".)

I think another way in which more children being born is civilizationally good and postpones AGI is that people with children are somewhat busy with children, and have their priorities shifted. A tail risk for you seems fine; a tail risk for your kids, less so. People think with clearer heads about risks to their kids, and so will be less likely to push for acceleration. Also, people having kids lowers the costs of other people having kids. We're in a kind of a crappy valley now with few people having kids, but if we bounce back to many people having kids, then the cost of childcare is lower due to huge economies of scale in having kids, and that helps a lot to motivate others.

Society has a terrible track record with this kind of technology

I don't think it makes sense for society to write off a possible future technology because it shares some general ideas with a past evil. In particular, reprogenetics applied to diseases can alleviate enormous amounts of suffering and death--think something like preventing many millions of cases of disease, if the technology is adopted at scale. If you don't believe this technical statement, we could try to clarify that (though there are existing papers that would do a fine job, and a more comprehensive FAQ hasn't been written yet). If we admit the large upside, that's already plenty of reason to consider the technology more in depth.

To elaborate, first I want to distinguish three things:

• Reprogenetics is a class of assisted reproductive technologies, some current (polygenic embryo screening, mitochondrial replacement therapy (kinda)) and some future (e.g. iterated meiotic selection, chromosome selection, iterated gene editing). (See "Methods for strong human germline engineering" for more detail.)
• Eugenics is an ideology, most active in the first half of the 20th century, that holds that there are Good and Bad traits--implicitly universally, and as judged by someone in power (the state, the doctors, the scientists, the elites, etc.).
• Genomic emancipation is an ideology that says it's good to make biotechnology to empower parents on behalf of their future children. More philosophically, it says: "The birthright of human spirits includes fuller self-unfolding via self-sovereignty over their own genomes.". (See "Genomic emancipation".)

Society does not have a terrible record with reprogenetic technology, because such technology has not existed until the past decade.

Society does have a terrible track record with the ideology of eugenics. One could generalize that, and say that society has a terrible track record with the general idea of somehow influencing heredity of future children.

However, this has a strange consequence of bundling genomic emancipation together with eugenics. Genomic emancipation negates the core idea of eugenics, and is therefore opposed to eugenics. Please see: "Genomic emancipation contra eugenics".

This bundling is, in this sense, incoherent. One reason you may nevertheless want to bundle these opposed ideologies together is something like this:

Unfortunately, it may not matter all that much what ideology you, Tsvi, have about reprogenetics. Reprogenetics is a tool that plays into the hands of the old and powerful ideology called eugenics. Even "genomic emancipation" takes on the basic ideas of controlling genes and reproduction in order to make "better people", and thereby lends support to eugenics. If lots of people support "genomic emancipation", this gives strong societal support to the idea that we can and should control the genes of future children; and this lends support to the idea that the government should police people's reproduction on behalf of the population, and that naturally leads to racist and coercive and otherwise creepy eugenical policies.

Or in your own words:

Of course you don't intend for things to go sour, and are advocating this for the best of motives - but the liberal eugenicists didn't intend that, either, and good motives don't guarantee good results.

I agree that this would be a bad outcome. I agree that it's a very legitimate concern. However, I want to largely refute this frame.

Of course, it's true that it doesn't matter too much what I personally do, at all. Insofar as it does matter what I personally do, though, my motives definitely matter, because they'll guide my actions and tell other people what to expect from my actions. But more to the point, I'm not saying "well, as long as I have pure motives, it'll all work out fine".

There is a fallacy here, but I'm not sure what the name for it is. It's something like the "action-counterfactual fallacy", or the "non-planning counterfactual fallacy", or the "non-agentic counterfactual fallacy", or "off-policy counterfactual" or "orphaned action" or something. In this fallacy, you argue against action X by saying "imagine action X is taken[, and everything else is held constant]; then the results would be poor". The problem is that this is the wrong counterfactual. The actual counterfactual should be more like "imagine action X is taken[, and also the rest of the plan that generated X is followed]; then such and such would happen". In this counterfactual, the results may be quite a lot better than in the fallacious counterfactual.

So, first of all, I'm not saying "I'll do reprogenetics, yes... but with good intentions!". I'm saying I'll do reprogenetics, and here's my philosophy, and also, here's a bunch of actions which I think will make this go well, which flow from that philosophy. Just a few examples of concrete actions:

• Lobby for "The principle of genomic liberty".
• Accelerate stronger forms of reprogenetics, which has various benefits including making it less expensive and therefore more widely accessible.
• Fund research to improve cross-ancestry generalization of PGSes, to make reprogenetics more equally accessible.
• Create innovation-positive ethical norms for reprogenetics, such as not pressuring parents into specific genomic choices; and enforce these norms (e.g. via a 3rd party auditor).

You wrote:

At the very least, I'd want to see a detailed argument for why this won't go wrong the way it's gone wrong in the past.

Allow me to make another distinction:

• What is likely to happen, in a future with reprogenetics?
• What are the different possibilities, for futures with reprogenetics?
• What can be done to change between such possibilities?
• What should be done to change between such possibilities?
• If we were to follow the best plan we could muster, what is likely to happen in a future with reprogenetics?
• If we were to follow the best plan we could muster, would the result be good in expectation? (What about in distribution, which is relevant to "Is this likely to be good for almost everyone, and awful for approximately no one?", which is relevant to constructing acceptable civil compromises.)

You're asking for an argument for why this won't go wrong. I do think it is likely to go well.

However, I'm rather less interested in making a detailed argument for that, compared to the activity of constructing better ways to make it go well. That's because:

• It seems like a more interesting and useful activity; it leads to ideas for how to make the world better.
• It doesn't make much sense to argue that it "would be good", due to the orphaned-action fallacy. What's the counterfactual? We can't even ask the right question without thinking through the positive vision in a bunch of detail.
• I expect constructing a good vision to be more persuasive than just arguing that it "would be good", and I do want to persuade more people to work on this. (Partly so that there are more people constructing the positive vision, working towards it if it's feasible, steering away from bad versions, and hopefully noticing if the bad versions really are too likely. Please see "Appendix: Why envision genomic emancipation?".)
• Predicting the future is hard. A lot of the reason I think reprogenetics is ethically acceptable, morally good, and altruistically a good bet, is not some detailed confident prediction about the future, but rather a mesh of partial predictions, ethics, and other considerations.
• More simply, the world is very complicated, so it's basically impossible to give a detailed confident argument for what will happen in the future. I've listed all the perils I could think of in "Potential perils of germline genomic engineering". Endlessly debating attempted predictions seems unproductive.
• If there's a plausible case that developing reprogenetics would be net bad for humanity, then of course that would argue strongly against proceeding without a clearer picture. I don't see such a case.

Here are some of the downside risks I see that seem plausible, AND related to eugenics, AND somewhat difficult to prevent through organized action (by the sector, by the public, and by liberal governments):

• There's a "soft eugenics" of social pressure, where people make genomics choices partly because of social approval or disapproval for specific genomic choices. I think this is kinda bad, but I don't think it comes close to flipping the sign of reprogenetics, unless it's very strong. I think this should be combatted, but doesn't come close to making a plausible case that reprogenetics would be net bad.
• There's imposition by authoritarian governments of policies about reproduction.
  • China, for reference, had mandatory contraception/sterilization as part of its one-child policy in the late 20th century and early 21st century, and employs forced sterilization as part of its persecution of the Uyghurs. Some countries, such as Saudi Arabia, have mandatory premarital screening for certain genetic conditions, with subsequent social pressure against marriage for double-carrier couples.
  • One could imagine a country that forces its citizens to use reprogenetics, and/or forces specific genomic choices for those who do use reprogenetics. This seems fairly plausible to happen in some forms. However, it seems to me that the plausible versions aren't that bad, and the worst versions aren't that plausible.
  • One possibility on the pareto frontier of plausible & bad according to me would be authoritarian regimes forcing their subjects to make their kids be more obedient, more conformist, less brave.
  • (Note, though, that they could already be doing that, as polygenic embryo screening has been available for about 5 years; time will tell.)
• Increased apathy or cruelty towards the frail, because "you should have been genetically modified". (I don't know how plausible this is to actually affect too much material policy as a consequence of reprogenetics, but that possibility doesn't seem wild, and would certainly be bad. As a touchpoint, prenatal screening for Down's syndrome seems to have become widespread, and it has decreased the number of Down's births; and also conditions for people with Down's seem to have gotten better over the same period. So in that one case, reprogenetics doesn't seem to cause apathy or cruelty towards the frail.)

Overall these seem bad if they happen, but not so bad and probable that they are at all in the ballpark of making reprogenetics net bad in expectation. Some of them might be alleviable. (E.g. we could push for international treaties against coercive reprogenetics. I wouldn't especially hold out too much hope on that, given that China's abuse of the Uyghurs hasn't provoked enough response to stop it; but on the other hand China did crack down on embryo gene editing in the He Jiankui affair, seemingly in response to international pressure.)

Some other possibilities seem also not very likely:

• For example: There's a fairly deep association between eugenics and racism. Does that also suggest reprogenetics is associated with racism?
• It seems like people who want to be racist are just going to be racist.
• Reprogenetics is about effects of genes, measured by trait differences between individual people; "race differences" for traits don't enter it at all.
• You could draw some sort of subtle line between things, like: if we're doing reprogenetics, then we're saying genes have some sort of biological effect on some of a person's traits; and so you think genes are real; and therefore you think race is real (??).
• But IDK, I just don't buy this causal arrow much. Would we also argue against clinical genetic screening (e.g. to direct testing and treatment) because thinking that genes are real is making people more racist? I mean, I don't want to be too dismissive, I have not looked into this specific worry. Maybe that causal arrow does exist? But I'm skeptical that it's a big arrow.
• I think eugenical ideology stems from high modernism and picks up genes as a tool, not the other way around. Reprogenetics in the framework of genomic emancipation is a tool for just that--emancipation--which is anti-authoritarian, anti-high-modernist, pro-pluralism.
• It would help me if you could name specific bad scenarios, or general principles, that you believe fairly confidently pose a serious risk here.

I think the sector of reprogenetics does have various responsibilities. It has responsibilities to:

• notice, think about, and work to alleviate and avert potential harms of reprogenetics;
• self-regulate to uphold the norms that would make reprogenetics beneficial for people and for humanity as a whole;
• stop if there's a major (fairly likely and quite bad) unanswered downside risk.

However, I don't view it as a responsibility of the sector to get everyone on board with reprogenetics before proceeding, or to police the behavior of the entire world with regards to novel reprotech.

For my ideas about how the sector of reprogenetics, and society at large, should comport themselves with regards to reprogenetics, please see "Some practical norms for good development of reprogenetics".

One might say:

It's all very well and good to have a whole plan. But you, Tsvi, aren't in charge of the world or society or the sector of reprogenetics. Just because you have these grand ideas, doesn't mean any of that will happen. There are forces beyond your control which may very well make this technology turn out to cause more harm than good.

To which I say, yes of course. I am not saying "everything's automatically perfect by default". I am also not saying "if only society did what I want, things would be great; therefore there's no problem with reprogenetics, just put me in charge". That's not the procedure I'm proposing. The procedure I'm proposing has these elements, inter alia:

• We (you, me, society, the public, scientists, policymakers, workers in the sector of reprogenetics) should envision what good futures with reprogenetics could look like.
• The sector of reprogenetics should constitute itself, and thereby take on its appropriate responsibilities as part of society and humanity. For example, it should:
  • Make and enforce professional norms of behavior (genetic counseling, marketing, publishing, scientific validation, follow-up studies, data privacy, etc.).
  • Put resources into research that will widen access (e.g. biotech innovation to make the tech less expensive).
  • Listen and respond (with explanation and action) to concerns that people have.
• These two things take a bunch of work and should get resources (funding and talent).

I think that with a high-effort, intentional version of this, the sector of reprogenetics actually can decide for itself to implement better versions of my and others's plans that will fulfill its responsibilities to humans and humanity, and do so in collaboration with governments and societies to implement broader structures (laws, social attitudes) that further build out a good future with reprogenetics. There are some forces beyond the control even of all of those constituencies put together, but I don't think that's a strong argument against doing this.

I think it wasn't a coincidence that liberal eugenics went so wrong.

(I take it you mean, early 20th century eugenics in liberal countries, rather than "liberal eugenics"?)

The theory overpromised and was unsound, especially as you got into more details. It also played to people's prejudices.

(True.)

Now, regarding this:

This seems to be repeating the exact pattern - overpromising, unsound on the details, playing into people's prejudices. Of course you don't intend for things to go sour, and are advocating this for the best of motives - but the liberal eugenicists didn't intend that, either, and good motives don't guarantee good results.

First, an overall comment: As I mentioned above, I think that this comparison is inapt. Comparing them because they both involve genes and reproduction is a poor comparison because that's not what makes them really distinctive. It's not the underlying engine of those behaviors. It's not that there's something about genes and reproduction that just makes people go crazy and want to forcibly sterilize people, deport all the brown people, etc. It's that there's a somewhat coherent ideology behind historical eugenics, and from that ideology flows much of that other behavior, in conjunction with associated attitudes such as high modernism / anti-pluralism. On that more salient dimension of comparison, genomic emancipation is anti-eugenics. Please see "Genomic emancipation contra eugenics".

Now on some particulars:

overpromising

Could you be more specific about the overpromising? Who is doing the overpromising (me in this post, other people somewhere, people in the future)? What is being promised, and what do you think is incorrect about that? To be clear, I think that:

• The field is quite vulnerable to overpromising, because statistical genetics is new, the results will take years to show up and will be noisy, and there's a lot of ways to misleadingly present claims.
• There probably has already been some degree of overpromising from polygenic embryo screening companies. At least one company has been accused of presenting its polygenic predictors as being considerably more accurate than they really are; and it seems common for gains to be expressed as "the difference between the highest and lowest risk embryo" for some disease--which is kinda misleading, compared to presenting the expected gain (i.e. lowest vs. average).
• Overpromising is quite bad for various reasons.
• Therefore it's important to only make accurate promises with appropriate uncertainties.

As with many other worries, this is also something to improve. I'm trying a bit to help the situation, mainly by trying to set up a 3rd party academic validator of PGS claims (though it sure would be nice if good scientists who could work on this didn't have to worry so much about being vaguely associated with eugenics). Eventually the sector should have self-regulatory orgs, professional standards, legal regulation, public commitments and norms, etc.

But I don't think this field has been especially bad in this dimension so far, compared to lots of other sectors. Lots of commercial sectors hype stuff up to investors, to customers, to potential hires, and to the public at large. I think that's generally bad (because generally deceptive) and I don't excuse it in reprogenetics, but I don't see how it's part of some pattern specifically in common with historical eugenics.

I think it is also harmful to underpromise about technologies. Don't forget the basic facts here--if strong reprogenetics were widely adopted, it would save at least hundreds of thousands of lives, probably more. (This is a somewhat far away hypothetical, but it's still important!) If you think this is overpromising, we may have a technical disagreement, which could be productive to clarify?

unsound on the details,

Same questions as above--who's being unsound and how? Be specific please.

playing into people's prejudices.

What do you mean by this? Who / what statements are playing into people's prejudices, and how?

Of course you don't intend for things to go sour, and are advocating this for the best of motives - but the liberal eugenicists didn't intend that, either, and good motives don't guarantee good results.

Hm. Are you sure they didn't get what they intended? Some of them jumped ship, though my impression is that a lot of that was due to the science turning out to be super wrong. A lot of them were excited with how things were going, and wanted it to go more! But I'm not sure what you're referring to, to be honest. Maybe you could point me to a source that gives some detail on this?

It seems to me that adding more high-IQ people to the "set of people thinking about AGI" has been tried in the past, and the effect was to accelerate AI capabilities

I'm curious what has been happening with young geniuses specifically; do they seem more or less accelerating AI capabilities compared to the older guard?

What you fundamentally need is dispositional/sociological, not mental horsepower.

I think you need both, by a long shot. Humanity is bottlenecked on ideas in so many ways in so many domains. Humanity is also bottlenecked on values / caring about the right things / being wise / etc.

But I would also say that humanity is specifically bottlenecked on having people who are both wise and very smart. We can at least unblock on "very smart".

I also think that if there were a lot more very smart people, we'd do a better job at supporting them.

That said, education is super important.

Also, I agree that wisdom specifically is important and interesting, and that we should investigate whether / how we could genomically vector for traits like that.

without serious effort at e.g. public red-teaming.

Yes, please send serious effort at public red-teaming. (Also send serious effort at public blue-teaming.) You can rest assured though, approximately no one from the community is working on this.

Here's my main efforts so far re/ red-teaming: https://www.lesswrong.com/posts/K4K6ikQtHxcG49Tcn/hia-and-x-risk-part-2-why-it-hurts https://berkeleygenomics.org/articles/Potential_perils_of_germline_genomic_engineering.html

extremely achievable with ordinary biological science.

I agree with this characterization, if we're taking a bird's-eye view of science. But to be clear, there are lots of unknowns here, such as how to do full in vitro gametogenesis, how to avoid genome degradation in cultured cells, what happens when you push a polygenic score way far out, how can chromosome selection be implemented, etc. Of course, there are also non-science / non-technological issues (legal, social).

And it's not like having healthier and smarter children is a thing people would be opposed to absent AI risk.

I definitely think there is a strong case for reprogenetics absent AI risk, and that's the case I mostly want to be making to society. In other words, I think it's mostly reasonable & good for society to not really account for AI risk when considering whether reprogenetics is a good idea, and instead evaluate it only on other grounds, such as whether it's good for the kids and whether society can handle that technology healthily. (In part because that consideration somewhat instrumentalizes children. It shouldn't be categorically forbidden to be motivated to have kids because of some outcome you want them to bring about, but it should be viewed with a lot of skepticism / worry / carefulness, especially when coming from society rather than parents and especially when it's a potentially overbearingly strong motivation.)

1. pre-RSI AGI are likely less aligned than enhanced humans

2. "Human intelligence enhancement could be related to data progress instead of algorithmic progress, as Beren once said about LLMs": what? When using data vs algo metaphorically to talk about humans, what's the human version of "data" and the human-version of "algorithm"? It's unclear how you map here..

Are you saying that native intelligence doesn't matter as much as education? Because I'm very sure that is incorrect; my intuition points strongly to many life outcomes pointing better for a 135 iq person vs a 100 iq person, all else being equal~ (and with the exception of mental illness/depression/burnout, which is a fraction of the population but far from 100%.

My basic case, available in the linked posts in some more detail and breadth, is:

1. By default, we have "very long" timelines because we don't have a strong reason to think AGI gets invented at some specific time.
2. To update strongly on that, we'd have to get strong evidence that we've gotten most or all of the ideas involved in making AGI.
3. People adduce the rise in capabilities as the main reason that they updated to confident short timelines.
4. But the rise seems largely explicable as ~imitation + a cherry on top. Or in other words, the inference to "we've found most of all of the ideas" is not supported by our observations
5. People adduce the utility of AI coding in AI R&D as a reason to expect fast future progress.
6. But this doesn't say much about how far is left; and I'm pretty skeptical of huge (like >10x, say) speedups on the research tasks that most matter for AGI capabilities progress.
7. So we should somewhat still be in the prior, though with some substantial update towards short timelines; but not way over to like "90% in 10 years" and similar. That's unjustified.

You write:

And why just scaling up pretraining more won't just work

Would you please read https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#The__no_blockers__intuition and maybe some surrounding sections?

People are not as loyal to the interests of other people as we would like, but we have no idea how to make an AI care even a tiny bit about people in a way that has a high probability of persisting after the AI has become super-humanly capable.