I've recently been having moderate success getting useful AI outputs on conceptual tasks by asking them to make a formal model of a domain that's hard to formally model, and giving it some blog posts or similar and saying "you should design your formal model to match these blog posts." Here's an example prompt that I found generated a useful response (at least superficially, idk if it will importantly affect my thinking medium-term):

"Make a simple formal model of alignment and how it might break, and a notion of scalable oversight in that formal model. This should be simple and concise. You should develop the formal model with the above [brief discussion of scalable oversight we already had in chat] in mind, and even more importantly, many posts from Paul Christiano and Jan Leike and [this one](https://www.alignmentforum.org/posts/hw2tGSsvLLyjFoLFS/scalable-oversight-and-weak-to-strong-generalization) about scalable oversight. Think long and hard to iteratively develop a model that's both very simple and captures the key points all this writing is talking about."

Here's one more:

"I want a formal theory of motivation generalization in AI, but I don't think it's very feasible. Try to give a simple formal model (built on sets and functions) that captures the core ideas from the og deliberative alignment paper, emergent misalignment in production RL, and the hugging face incident (where in an unusual context, the AIs did egregiously misaligned things despite usually being roughly good instruction followers).

Your answer should be short."

I put the full responses in this doc if people want to see.