Let's imagine a Solomonoff-style inference problem, i.e. our inference-engine receives a string of bits and tries to predict the next bit.

Here's an anti-Occamian environment for this inference problem. At timestep t, the environment takes the t bits which the inference-engine has seen, and feeds them into a Solomonoff inductor. The Solomonoff inductor does its usual thing: roughly speaking, it finds the shortest program which would output those t bits, then outputs the next out-bit of that shortest program. But our environment is anti-Occamian, so it sets bit t+1 to the opposite of the Solomonoff inductor's output.

That's an anti-Occamian environment. (Note that it is uncomputable, but still entirely well-defined mathematically.)

One example of an "anti-Occamian hypothesis" would be the hypothesis that the environment works as just described - i.e. at every step, the environment does the opposite of what Occam's razor (as operationalized by a Solomonoff inductor) would predict.

An agent which believed the anti-Occamian hypothesis would, for instance, expect that any hypothesis which did a very good job predicting past data would almost-surely fail at the next timestep. Of course, in a world like ours, the agent would be wrong about this most of the time... but that would only make the agent more confident. After all, it's always been wrong before, so (roughly speaking) on the anti-Occamian hypothesis it's very likely to be right this time!

Again, note that this example is uncomputable but entirely well-defined mathematically. As with Solomonoff inductors, we could in principle construct limited-compute analogues.

The standard example of a system which (maybe) behaves in this anti-inductive way in the real world is a financial market.

I think this is similar to the 2010 post A Proof of Occam's Razor? ...which spawned 139 comments. I didn't read them all. But here's one point that came up a couple times:

Let N be a ridiculously, comically large finite number like N=3↑↑↑↑↑3. Take the N simplest possible hypotheses. This is a finite set, so we can split up probability weight such that simpler hypotheses are less likely, within this set.

For example, rank-order these N hypotheses by decreasing complexity, and assign probability $2^{-n}$ to the n'th on the list. That leaves $2^{-N}$ leftover probability weight for all the other infinity hypotheses outside that set, and you can distribute that however you like, N is so big that it doesn't matter.

Now, simpler hypotheses are less likely, until we go past the first N hypotheses. But N is so ridiculously large that that's never gonna happen.

I think it depends on the size of your outcome-space? If you assume a finite (probably generalizes to compact?) space of outcomes, then your argument doesn't really go through. But a lot of things seem better modelled with infinite outcome spaces, so in that case your argument seems to go through, at least under conventional formalisms.

It seems like Occam's Razor just logically follows from the basic premises of probability theory.

This turns out to be the case for countable probability spaces, like Turing Machines, due to Qiaochu Yuan's comment:

https://www.lesswrong.com/posts/fpRN5bg5asJDZTaCj/against-occam-s-razor?commentId=xFKD5hZZ68QXttHqq

It seems like Occam's Razor just logically follows from the basic premises of probability theory. Assume the "complexity" of a hypothesis is how many bits it takes to specify under a particular method of specifying hypotheses, and that hypotheses can be of any length.

I think the method can't be arbitrary for your argument to work. If we were to measure the "complexity" of an hypothesis by how many outcomes the event set of the hypothesis contains (or by how many disjuncts some non-full disjunctive normal form has), then the more "complex" hypotheses are less likely.

Of course that's an implausible definition of complexity. For example, using conjunctive normal forms for measuring complexity is overall more intuitive, since $A\wedge B$ is considered more complex than $C$, and it would indeed lead to more "complex" hypothesis being less likely. But universally quantified statements (like laws of nature) correspond to very long (perhaps infinitely long) CNFs, which would suggest that they are highly complex and unlikely. But intuitively, laws of nature can be simple and likely.

This was basically the problem Wittgenstein later identified with his definition of logical probability in the Tractatus a hundred years ago. Laws would have logical probability 0, which is absurd. It would mean no amount of finite evidence can confirm them to any degree. Later Rudolf Carnap worked on this problem, but I'm pretty sure it is still considered to be unsolved. If someone does solve it, and finds an a priori justification for Ockham's razor, that could be used as a solution to the problem of induction, the core problem of epistemology.

So finding a plausible definition of hypothesis complexity, for which also Ockham's razor holds, is a very hard open problem.