The AI is asked mathematical questions relating to the big unsolved problems of physics: quantum mechanics vs. general relativity, the structure of space-time at the Planck scale, dark matter, dark energy, and so on. Its answers enable the easy construction of superweapons that convert matter directly into energy; or make anything radioactive decay instantly to its end state; or collapse the false vacuum; etc. Shortly thereafter, everyone dies.
ETA: see also.
The problem here is that there can be questions whose answers are dangerous to know.
How are you imagining making this thing? If you're training an optimizer that you don't understand, this can lead to pitfalls / unintended generalization. If it just pops into existence fulfilling the spec by fiat, I think that's a lot less dangerous than us using ML to make it.
Since the question is about potential dangers, I think it is worth assuming the worst here. Also, realistically, we don't have a magic want to pop things into existence by fiat so I would guess that by default if such an AI was created it would be created with ML.
So lets say that this is trained largely autonomously with ML. Is there some way that would result in dangers outside the four already-mentioned categories?
AI becomes trusted and eventually makes proofs that can't otherwise be verified, makes one or more bad proofs that aren't caught, results used for something important, important thing breaks unexpectedly.
Let's say we limit it further so all proofs have to be verifiable within some reasonable complexity limit. In such a case, we wouldn't need to trust it. What then?
Posting this on behalf of user PrimalShadow on the AstralCodexTen Discord:
I responded with the suggestion of what you might call "paperclipping" (it deciding it requires some astronomical amount of compute to solve the problem, so takes over the world, or maybe even the galaxy/beyond, to maximize compute), which PrimalShadow noted already falls into "The AI somehow learns about the outside world". He also pointed out that the AI wouldn't need to be trained on non-mathematical data, which would presumably reduce any risk of that happening.
Neither of us could easily think of a scenario where this trivially goes wrong, and we're both curious what the potential risks are from the perspective of more experienced people in the field.