I think that people overrate bayesian reasoning and underrate "figure out the right ontology". 

Most of the way good thinking happens IMO is by finding and using a good ontology for thinking about some situation, not by probabilistic calculation. When I learned calculus, for example, it wasn't mostly that I had uncertainty over a bunch of logical statements, which I then strongly updated on learning the new theorems, it was instead that I learned a bunch of new concepts, which I then applied to reason about the world. 

I think AI safety generally has much better concepts for thinking about the future of AI than others, and this is a key source of alpha we have. But, there are obviously still a huge number of disagreements remaining within AI safety. I would guess that debates would be more productive if we more explicitly focused on the ontology/framing that each other are using to reason about the situation, and then discussed to what extent that framing captures the dynamics we think are important. 

I think it would be good if more people say things like "I think that's a bad concept, because it obscures consideration X, which is important for thinking about the situation". 

Here are some widely used concepts I think are bad and I wish became less load bearing in AI safety discourse: 

• "Fast" and "slow" takeoff; takeoff speeds in general. I think these concepts are very unclear and not super natural. There are various operationalizations of these (e.g. Paul's "slow takeoff" = a gdp doubling over the course of the 4 years before the first single year that gdp doubles). This is obviously arbitrary, and I don't see why world's that meet this definition are worthwhile reasoning about separately from worlds that don't meet this definition. I also think its easy to smuggle in lots of other correlations here, like slow takeoff = people are woken up, or slow takeoff = alignment is much easier, etc. 
  • I feel more excited about talking about things like "I think milestone X will happen at date Y", or "My median is that milestone X and Z are Y time apart". For example, I think the concepts of automating coding, automating all AI research, automating ~the whole economy, and increasing earth's energy output 1000x are all useful capability milestones, and it's useful to talk about these. 
• "scheming". I think the definition of scheming is pretty unclear, and changes a bunch depending on the context. 
  • Under some definitions I believe that the AIs are always going to be scheming, under others, it seems kind of narrow and unnatural. 
  • I somewhat prefer the concepts from the "alignment over time" box in AI 2027.
• "gradual disempowerment" 
  • I think that this conflates a bunch of scenarios / threat models together, some of which don't make sense, some of which do make sense, but I don't think that the solutions are very related. 
  • Gradual disempowerment is often presented as not requiring any misalignment on behalf of the AIs. If there are AIs that are aligned with any human principals, we get a situation where, even without any coordination, AIs compete on behalf of their principals, and then give the principals whatever surplus they are able to produce. 
  • Given that framing, we can now talk specific threat models. For example, maybe their is no surplus: warfare/competition eats away all the additional resources, and space is consumed purely by the optimal self replicators. Alternatively, maybe the AIs weren't actually acting in the interests of humanity. Finally, maybe the process of competing hard was existentially catastrophic early on, e.g., maybe it resulted in the oceans being boiled (and humans didn't take appropriate countermeasures), resulting in extinction. 

Note: I'm not saying that all concepts that I think are bad can be saved or crystallized into good concept, often the best idea is to just forget about the original concept and use better/crisper concepts. 

I should also maybe give some concepts I think are generally good: p(doom) from AI takeover, timelines, x-risk, s-risk, recursive self improvement, internal/external deployment, AI control, AI alignment. I don't think any of these concepts are fully crisp, all have somewhat important edge cases, but I think they are good enough abstractions to be very useful for thinking about the future.