otto.barten

otto.barten's Comments

Looking for non-AI people to work on AGI risks

Also another thought. (Partially) switching careers comes with a large penalty, since you don't have as much previous knowledge, experience, credibility, and network for the new topic. The only reason I'm thinking about it, is that I think AGI risk is a lot more important to work on than climate risk. If you're moving in the opposite direction:

1) Do you agree that such moving comes with a penalty?

2) Do you think that climate risk is a lot more important to work on than AGI risk?

If so, only one of us can be right. It would be nice to know who that is, so we don't make silly choices.

Looking for non-AI people to work on AGI risks

Hi Brian, thanks for your reply! I think we would not need very special qualifications for this, it's more a matter of reading up on the main status of AI and safe AI, cite the main conclusions from academia and make sure they get presented well to both policy makers and normal people. You say you'd expect countless others to want to work on this too, but I didn't find them yet. I'm still hopeful they may exist somewhere, and if you find people already doing this, I'd love to get in contact with them. Else, we should start ourselves.

Interesting observation! I'm thinking that your second front is especially interesting/worrying where AI improvement tasks are automated. For a positive feedback loop to occur, making AI get smarter very fast, many imagine an AGI is necessary. However, I'm thinking, what's improving AI now? Which skills are required? I think it's partially hardware improvement: academia and industry working together to keep Moore's law going. The other part is software/algorithm improvements, also done by academics and companies such as Deep Mind etc. So if the tasks of those researchers would be automated, that would be the point at which the singularity could take off. Their jobs tend to be analytical and focused on a single task, more than generically human and social, which I guess means that AI would find them easier. That in turn means the singularity (there should be a less scifi name for this) could happen sooner than AGI, if policy doesn't intervene. So also a long winded I agree.

So how should we go about organizing this, if no one is doing it yet? Any thoughts?

Thanks again for your reply, as I said above it's heartening that there are people out there who are on more or less the same page!

Looking for non-AI people to work on AGI risks

Hi WH, thank you for the reply! I find it really heartening and encouraging to learn what others are thinking.

Could you explain what hardware you think would be needed? It's kind of the first time I'm hearing someone talk about that, so I'm curious of course to learn what you think it would take.

I agree with your point that understanding risks of AI projects is a good way of framing things. Given the magnitude of AGI risks (as I understand it now, human extinction), an alarmist tone of a policy report would still be justified in my opinion. I also agree that we should keep an open mind: I see the benefits of AI, and even more the benefits of AGI, which would be biblical if we could control the risks. Climate adaptation could indeed be carried out a lot better, as could many other tasks. However, I think that we will not be able to control AGI, and we may therefore go extinct if we still develop it. But agreed: let's keep an open mind about the developments.

Do you know any reliable overview of AGI risks? It would be great to have a kind of IPCC equivalent that's as uncontroversial as possible to convince people that this problem needs attention. Or papers stating that there is a nonzero chance of human extinction, from a reliable source. Any such information would be great!

If I can help you by the way with ideas on how to fight the climate crisis, let me know!