Let's consider a simple universe consisting of an ideal gas inside a box (which means a number of massive points behaving as prescribed by Newton's physics with completely elastic collisions).
After the system reaches the thermodynamic equilibrium you could ask if it will stay forever in this equilibrium state, the answer turns out to be negative: Poincarè recurrence Theorem applies and we can be sure that the system will go far from equilibrium again and again given a sufficiently long "life" for this universe. Therefore entropy in this universe will be a fluctuating function like this:
If you are patient enough you would see
- countinuous micro-fluctuations
- occasional slightly bigger fluctuations
- rare "not-so-small" fluctioations
- extremely rare medium-big fluctuation
- unconceiveably rare super-big fall of entropy like this
The interesting thing about these fluctuations is that their graph would look almost the same if you reverse the time (and this is because the underlying physics laws followed by the particles are time-symmetric). If you started in a special low-entropic configuration (for example the gas was concentrated) your reference to distinguish the past from the future could be the increasing of entropy at least at the very first moment until you reached the equilibrium. However if you wait long enough after the equilibrium is reached you would see the graph reaching again any possible level of low entropy so there is really nothing special in the low entropy at the beginning.
So this toy universe is actually symmetric in time, there is nothing inside it that would make a particular direction of time different from the other.
Now let's consider our real actual universe. Like the toy universe it seems we have again time-symmetric laws of physics applied in a far more complex system of particles. Again if we are asked to distinguesh the past from the future we rely our decision on the direction of increasing entropy but like before it could be misleading: maybe our universe is fluctuating too and we are just inside a particular super-big fluctuation, so there is nothing special about low entropy that forces us to put it in the "past": it will eventually happen also in the future, again and again. Furthermore if we are inside a fluctuation we have no way to tell if we are inside the decreasing part rather than the increasing part of the fluctuation: they would look very similar if you just reverse the time arrow. You could think you "see" that entropy is increasing but your perception of the time depends on your memory and if entropy is decreasing maybe your memory is just working "backwards", so your perception is not reliable, we have just the illusion to see the time flowing in the way we think (and actually our memory it is not really a "memory" because it is not even really "storing" information, it is anticipating the future that we think is the past).
What is the probability that we are in the entropy-decreasing part rather than the increasing part of the process? As far as we can tell it should be 50%, given the information that we have: the two parts have the same frequency in the life of the universe. So it is equally possible that the world is behaving in the way we think it is and that the world is behaving in a super wild and improbable way: the heat is flowing from cold to hot objects, the free gasses are contracting, the living being are originating from dust and ending inside the woumb of their mother and all this sequence of almost absurd events is happening just by chance, an incredibly improbable combination of coincidences. So we should conclude that this absurd scenario is equally likely to be true as the "normal" scenario where entropy is increasing. This could seem a crazy though but it seems logically reasonable according to the argument we considered above. Isn't it?