On the last one of your three examples, I feel that ‘mesa optimizers’ is another regrettable example of the forces of linguistic entropy overwhelming any attempts at developing crisply stated definitions which are then accepted and leveraged by the entire community. It is not like the people posting on this site are incapable of using the tools needed to crisply define things, the problem is that many do not seem very interested in ever using other people’s definitions or models as a frame of reference. They’d rather free-associate on the term, and then develop their own strongly held beliefs of what it is all supposed to be about

Yes.. clarity isn't optional.

MIRI abandonned the idea of producing technology a long time ago , so what it will offer to the the people who are working on AI technology is some kind of theory expressed by n some kind of document ..which will be of no use to them if they can't understand it.

And it takes a constant parallel effort to keep the lines of communication open. It's no use "woodshedding" , spending a lot of time developing your own ideas in your own language.

There's no strong reason to think the brain does everything with a single algorithm.

If the value is in saving lives, then the two are not orthogonal.

It's not just a case of any two agents having fuzzy approximations to the same world view. In the least convenient case, agents will start off with radically different beliefs, and those beliefs will affect what they consider to be evidence, and how they interpret evidence. So there is no reason for agents to ever converge in the least convenient case .

Aumann's theorem assumes the most convenient case

Former boss: you don't need comments, you can just look at the code and see what it does..

Me: you need comments to tell you why it does it...

What is truth? Rather than get into philosophical debates about this one, let’s use a reasonable working definition that by truth we mean “accurate predictions about our experiences

I would have though that definition was less impacted by the PotC than most. You can check directly that predictive theory is predicting, do you don't need apriori correctness.

This means that you’re going to need some skill at reasoning about non-truth-seeking agents. But the human brain is kinda bad at thinking about minds not like our own. A highly effective way to overcome this is to build cognitive empathy for others by learning to think like them (not just to model them from the outside, but to run a simulated thought process as if you were them). But this requires an ability to prioritize something other than truth because the agent being simulated doesn’t and because our brains can’t actually firewall off these simulations cleanly from “our” “real” thoughts (cf. worries about dark arts, which we’ll discuss shortly

If you're capable of firewalling off your model of non-truth-seekers, I don't see a further problem. You don't have to stop prioritising truth in order to model non-truth-seekers because prioritising truth requires modelling non-truth-seekers as non-truth-seekers.

the “critical rationalists” (who are a contemporary movement that involves David Deutsch, the “taking children seriously” people, and some larger set of folks who try to practice a certain set of motions and are based out of the UK, I think)?

Critical rationalism is basically the scientific philosophy of Karl R. Popper. An Austrian, he relocated to the UK in the 30s for similar reasons to Sigmund Freud's. So CR ended as being a kind of UK thing, despite having its roots in the Vienna Circle. (It also has a following in Oz and NZ, but not so much in the states).

It's not particularly contemporary, therefore...more of a version 2.0 of logical positivism. I don't know of any specific connection between mid 20th century UK atheism/humanism and CR...but it would be strange if there were none.

There were a lot of developments in philosophy-of-science post WWII -- Kuhn, Feyerabend , etc -- and CR somewhat faded from fashion until Deutsch revived it with his popular work in the 1990s. (Although Popper's falsificational criterion for science remains popular among working scientists).

Deutsch's (and other subsequent) versions of CR got entangled with Austrian economic, libertarianism ,and various other minority beliefs.

If you want to determine the balance of evidence, by all means do so, but you can't do that by completely disregarding the alternative explanation.

There was a time when thus place was all about the avoidance of bias.

You are responding as though I said something like "physics doesn't work at all", when I actually said it works via idealisations and approximations. To talk of Effective Field Theories concedes my point, since EFTs are by definition approximations .