Generally, in such disagreements between the articulate and inarticulate part of yourself, either part could be right.

Your verbal-articulation part could be right, the gut wrong; the gut could be right, the verbal articulation part wrong. And even if one is more right than the other, the more-wrong one might have seen something true the other one did not.

LLMs sometimes do better when they think through things in chain-of-thought; sometimes they do worse. Humans are the same.

Don't try to crush one side down with the other. Try to see what's going on.

(Not going to comment too much on the object level issue about AI, but, uh, try to be aware of the very very strong filtering and selection effects what arguments you encounter about this. See this for instance.)

Think about your beliefs' consequences a lot and in detail. Personalize your imagination of the outcomes, imagining consequences to you and to others as vividly as possible.

I'm pretty sure that's the mechanism (after working on the neuroscience of motivation for a long time), but I'm not sure you should actually do that for existential risk from AGI. I work full-time on AGI risk, but I really enjoy not fully feeling the consequences of my beliefs WRT doom (I give it roughly 50% since so much of the logic is poorly worked through thus far). Would I work a little harder if I was more terrified? Probably for a while, but I might well burn out.

One possible solution is to do the opposite type of motivation: think of the consequences of succeeding in aligning AGI (or avoiding other dangers). Think about them in as much detail and as frequently as you can. The imagination has to be vivid enough to evoke emotions; tying those emotional experiences to the concepts is what makes you feel your beliefs.

Imagining a vast number of humans, transhumans, and sentient AIs enjoying their lives to the fullest and enjoying activities we can barely imagine is a way more fun way to motivate yourself.

Challenge yourself to imagine how much incredible fun people might have if we get aligned superintelligence (I like to assume near-perfect simulations, so that people can have lots of challenges and adventures without getting in each others' way, but there are more mundane ways to have immense fun, too)

I'm not sure if it's as effective, but for your own sake I'd recommend that over imagining how bad failure would be.

CFAR's approach to the problem was internal double crux. If internal parts disagree and have different beliefs, internal double crux is a way to align them.

Leverage Research developed with Belief Reporting another approach to deal with the issue.