Look, let me explain my perspective, here, then. The discussion took place on 3 entirely different 'epochs' (one 7 months ago, then another 6 months ago), and between one of these you directly told me to take a different angle; what you perceive as me being "determined to show that the post is fractally wrong" is, to me, just a result of rereading the full thing and finding different problems.

I will concede that my first approach was misguided; I was more appalled by the combination of title and closing thesis as not properly discussing the cons, and thought the useful thing to post for others to read was, roughly, the costs related to going down this road (incoherence, that is, credence-numbers that are not obtained by Bayes updating and as such do not have the advantageous properties one purportedly uses and defends Bayes numbers through, and of the kind that is hard to repair or argue is 'approximately' coherent in some implicitly-Bayesian sense) and the many unmentioned alternatives. I still think a better article would do both of those; I do not think this would make it too technical, especially the former.

Then you told me the thing has 'specific rules'; you specifically used the word 'rules'. I went and looked! That's been the attempt, here, with what you call a 'different abstraction'. My conclusion so far is that I have found no viable, good formalization of this idea; it either collapses back in the coarse model (insofar as one wants to keep coherence) or it has no properties that I can state as good or bad (if one allows the marginal likelihood to shift away). I think that the non-technicality of the post has led to an idea that is unsalvageable, at least as a procedure with 'rules', in the current form. If you wanted me to discuss it as a practical, pragmatic device for thinking about things in a pseudo-Bayesian way, then, I think I get to pout at the fact that you pointed me in the exact opposite direction. (Not only that, but, when I stated my understanding of the thing, you took it upon yourself to agree that it was, at least roughly, correct!)

So, okay. I will put away the snark and the rhetoric and just state my problems with the thing directly. Afterwards I will elaborate on what I believe can be patched, what I believe lacks nuance and warning as presented (of the kind that you have displayed in these comments after), and what I cannot help but conclude is just misguided, or at least requires substantial refinement. For whatever reason I think it is best to express these in pairs. Apologies for length.

Coherence is hard. Doing a full analysis of any problem, even simple, informal checks, while staying fully within the Bayesian formalism is sometimes basically impossible.

However, it is not impossible. Sometimes, with a little difficulty, it can be done. This often requires care; it sometimes requires accepting things that seem unintuitive or even nonsensical. This is because most of the results that point to Bayesianism as the thing to do, the reasons why one performs Bayesian updating and not Dempster-Shafer updating or Jeffrey updating or ad hoc numerical reasoning, usually come in the form of 'if and only if's; the Dutch book converse, the CCT converse, the accuracy theorems, Cox's theorem, forecasts-as-martingales, all of these uniquely identify probabilities-as-credences or Bayes updating as, specifically and uniquely, the thing to do; if you end up doing something else, you lose the properties instantly.

This difficulty means that, often, approximating coherence is essentially required.

If and only ifs are very fragile things; you step away from the formalism in one subtle way and all of these things have suddenly broken into pieces somewhere. What you must ensure, then, is that most of the pieces are still there. Sometimes this can be done formally; I mentioned logical induction; there it is shown that one is only boundedly Dutch-Booked in a relatively benign way. Safebayes is a more pragmatic one; it is very clear to see how that is very approximately Bayesian. Empirical Bayes as approximate hierarchical Bayes.

Sometimes the approximation looks right, and becomes unsalvageable very quickly, in some mathematical sense. This is the case for many objective priors; everyone can nod at the univariate Jeffreys' rule, but not even Jeffreys defended the multivariate one. You may fondly remember the right-Haar prior, but the left-Haar inspires little fondness. Very classically, nobody puts up a fight to defend flat priors anymore. And so on. I tried putting your procedure in some formalizations and this seemed to be the case; but you (now) reject them, so I will not reiterate.

Sometimes, there is no formal procedure to analyse the proximity of.

Other times, the approximate-incoherence is incurred from some (sometimes necessary) pragmatic data-procedure. You change the prior after the sampler runs for too long, you pick the best model out of a bag after looking at a predictive score, you throw out the model because the predictions look kind of bad; nobody (in the 21st century) would dispute that these things happen.

However, what is missing in your post and not missing whenever Bayesian statisticians discuss these things is that when you do this you are treading very dangerous ground. So, generally, the best of them (which is to say, Gelman's cohort, Stephen Walker, Rubin, etc), if one cannot work out some alternative justification or method to do these things that is still within the formalism (as with Gelman in the model expansion cases, and in Rubin's case something as trivial as randomization), at the very least you run a bunch of checks to see that you are really not shooting yourself in the foot somewhere.

This is because, when you do abandon coherence in a way that does not give you some kind of 'bounded' error, even in some conceptual sense (as in your case, where the marginal likelihood shifts arbitrarily; I will reiterate that the problem is not the prior becoming more detailed, but the marginal likelihood shifting alongside it in a way that does not reduce the model back to it), you really are playing with fire. The numbers obtained by such a method do not have the philosophical clarity of the numbers obtained with Bayesian updating, even if you could imagine that you might have obtained the numbers through some Bayesian update that was not performed.

I understand that you may raise an eyebrow at this, but, the analogy with p-hacking is relevant: you do a little tweak here or there, in ways that might make a lot of common sense, hell, sometimes, it really would be better if you had ran the analysis you wish you had ran after you saw the data; and, really, if the null your tweak led to be rejected really was wrong, then one may feel some solace that no damage was done. But, still, the '0.05' number just no longer means a 0.05 error rate; the number has ceased having a meaning. You may run some garden-of-forking-paths analysis that might elucidate what that number might mean, but it is no longer the p-value it purported itself to be.

Besides, frankly, sometimes you have to put common sense and other statistical properties over following Bayesian edicts directly.

So, lacking a formalization through which to analyse approximation quality and observing that the procedure can output ultimately arbitrary results (thus even a conceptual approach to approximation seems hard), since they are no longer clean Bayes-rule probabilities obtained from a prior and posterior update, what are they for?

It is easy to make this kind of procedure look good if one picks out examples where it happens to reach the correct answer (when the null is false and you performed a terrible analysis, p-hacking can look absurdly, genuinely convincing!). I don't know about you, but, for me, and I think for most statisticians, we like to look at what happens if the person doing it is maybe a little tendentious someplace; maybe not too disciplined here or there. A statistical (or a Bayesian) justification can often serve as a little "certificate of formal correctness" for people who will then no longer squint when they do some things that they really should squint at.

So, how does this procedure fare in this respect? I honestly think that the infamous "Bayesian arguments for God" are, frankly, not too far away from this; you could probably put them in exactly this language, or derive them in this way. "I first took a prior and likelihood, did an update, it did not seem sensible to me; I then found the little tweak to the model that made it make sense with my intuition, so now I have obtained a Bayesian justification for my belief"; I think you know how bad this can be. I am not saying this to imply that the idea fares as poorly as that in general, I imagine in most situations it acts moderately more benignly, but that it is a natural thing to reach for, and easy to imagine that you are doing nothing wrong through, regardless of how right you are (or aren't).

Of course, you say that things should only be done in the right way, in the post. You do not, I think, specify how to do this; honestly the bisection you performed in the aliens example would look pretty tendentious to an aliens-believer, if you imagine that a few hundred years from now we discover that aliens really are real I think you know that the toy example looks pretty bad. A demonstration of a good method survives the particular example being wrong; one can at least be wrong gracefully, or at least where the flaw lies can be shown to subside in the specifications of the method.

In any event, this is a good model for something that is encountered in practice.

I think that you are right, and indeed you are right that people do some version of this, often. But, one virtue I can name of the Bayesians of the 21st century at least is that they do not do this, typically; if their model was wrong, they are fine eating the bullet, without performing any rigmarole that puts their objectivity into question, they are fine with putting "money on the table". Here, you salvage a broken inference by a method that simply would not convince a skeptic. But, the problem that you salvage lies in the likelihood, not in your prior; just allow yourself to observe some more data and you'll be fine, or split the sample first, or cautiously observe through a prior predictive check that your model allows no information to be gained regardless of what happens, or in fact just obtain a better prior (I believe witness unreliability to be the Hume thesis on miracles in the 1700s, so it can certainly be prior to the data in this example to consider it).

Ultimately, when I looked for rules, all I found was the rough sense that the posterior should match one's intuition. Of course intuition is often right and good, possibly often better than our Bayesian calculations. But, ultimately, I think that performing this kind of calculation to affirm one's intuitions is at best just an elucidation of at least part of why you think you think the things you think; at worst, and to be frank, very typically in this framing at least, a way to smuggle vague intuition as formally justifiable insight. The best cases of Bayesian inference are when you perform it and convince yourself out of an intuition that was actually wrong; spiritually, at least, this seems to basically prohibit this, there is always an out, always a way to slice up the likelihood to get numbers that you are more concordant with.

I think I will finish on that note. On how the idea can be patched: As Mateusz mentioned earlier, having the partition be data-independent solves the coherency issue entirely (qua Savage). Or, at least, a formal justification that binds one not to change the likelihood too much. Or, in some sense an argument that one will not easily mislead themselves, that one will not be able to convince themselves of false conclusions as easily as true ones, or a sensitivity analysis of some kind that shows, okay, maybe your new flimsy post-hoc model is not too flimsy. Or, as a last resort, if you want a number for an intuition, you can elicit a full posterior from your intuitions via all the other methods in a way that doesn't make it look like reasoning was performed when it really wasn't. Mostly, if the idea has formalizable value, it ought to be formalized, at least partially; if it can't be, then it should at least be argued that one does not easily mislead themselves through it.

On how the post can be improved, I already mentioned a fair few. If you think that your last paragraph is really only specifically about the toy aliens example and nothing else, I think it should be stated, since it is very unusual for articles to end on a note about the examples rather than what they illustrate. I also think alternatives and risks should be mentioned, if one cannot improve the idea as above to avoid them; a post on a heuristic is useless without showing where and how the heuristic fails, otherwise one falls off cliffs. Reframe the post as being about changing your model post-data, since that's what actually changes the inference, not the prior (the prior changing is the most benign part of this, ultimately; it just opens the door to an entirely different model, which does not match on the marginal).

Bunglesome to respond to 'you are tendentiously trying to show this is fractally wrong' with the longest reply yet, but, you put me between Scylla and Charybdis by asking me for clearly stating a full, coherent position in as neutral a light as possible. Would've written a shorter letter, etc.

Addendum: Possibly some of this comes down to a difference in intuition: you imagine your procedure as mimicking a clean, contiguous cut of a 1d curve; there it can at least kind of be imagined what a sensible cut may look like. Insofar as one applies this procedure to questions like your example, I am imagining that the cut is really happening on some infinite-dimensional, abstract space - God only knows what reasonable cuts look like, I have no idea if my particular one is random, tendentious, an improvement or a bias. Certainly I trust no-one else's.