Daniel Brach-Neufeld

Don't deontologists believe you're deontologically barred from destroying the world? You seem to have given an example where deontologists and consequentialists would disagree about the moral course of action, and then stated that obviously you should do the thing dictated by consequentialism because the consequences will be better. Look, I'm inclined to agree with consequentialism, but there are deontologists. They believe you shouldn't pull the lever in the trolley problem even though if you do pull it fewer people will die. Immanuel Kant really believed... (read more)

Yes, sorry, when I say "we'll have to see if such things emerge," I mean "it's not obvious to me that this is a big threat. I don't know if there are likely to be non-superintelligent rogue systems attempting to deploy bioweapons. It's not obvious to me that the reasons the rogue AIs commit cyberattacks would cause them to engage in bioattacks, or that there would be other reasons that misaligned AIs would use bioweapons not as part of a takeover scenario. AI-generated plagues may be a capability the AIs would have and not use before superintelligence." I ... (read more)

A pandemic might be the method of killing most humans that requires the least intelligence, but killing a lot of humans may not be the takeover step that requires the least intelligence. It seems possible that an AI becomes capable of killing much of humanity but lacks the physical infrastructure to ensure its own survival (which is likely to be an instrumental goal for fulfilling whatever weird terminal goals it has).

I can imagine a scenario where the AI is smart enough to make a deadly virus but where that's the easy part. The hard part is getting enough... (read more)

I think you're right that the capabilities we've seen so far have in LLM base models have mostly been emulating humans.

If I had to guess why (as someone who doesn't have a technical background in these things and so has low confidence in his conclusions), I'd guess that the reason we haven't seen a lot of growth in other stuff is because the tokens that require a model of the world (rather than a model of written text and human writing style) are a relatively small percentage of the tokens they're trying to predict, even though it's a large amount of info... (read more)

One thing to note is that the human-generated training data contains facts about the world not thought up by humans and correctly predicting it would require understanding possessed by no human. The model might be trained on a report of a clinical trial which says "the subject was administered 5mg of [drug] and was observed to ___." What happened? Did they recover? Develop a side effect? Have a complication? No human or collection of humans thought up the answer, they just observed the world, but the AI in training needs to accurately predict what they obs... (read more)