@Elizier

"I think you are confusing knowing that a system will perform arithmetic, with the system actually performing arithmetic. The latter does happen sometimes, despite all fallible assumptions."

I think you didn't understand my argumentation; when you say that a physical system does perform arithmetic, then your theory of arithmetic is wrong as soon as you have a contradicting result.Therefore the system is not allowed to perform arithmetic sometimes, but it is required to do it always!

Let's consider this: I find a machine I don't know anything about it. I soon find out it has two input dials and what looks like an output register.
By experimenting with the inputs and noting the outputs I found out that the inputs are presumably decimal numbers and the output looks like the arithmetic sum of these numbers.
I say: 'Oh, looks like it adds two numbers.'; now I'm using this machine many many times, until one day where the result isn't the arithmetic sum (let's assume there is no overflow).

'Bugger, seems this machine is broken...'

Now, the 1M $ question: 'At which point did the machine got broken?'. Did it get broken exactly at the point when it printed the wrong result? But what if the inner workings had the defect way before? And it only prints 'wrong' results for specific inputs?

You clearly don't want to question your theory of arithmetic (because your theory doesn't have any contradictions). But let's assume that the creator of this machine didn't want it to perform addition, but he wanted it to calculate the Foobar-value of two 'numbers'. The Foobar-value looks like addition for a majority of values, but for some combinations it's something completely different.

Of course you can examine the inner workings of the machine; but if you don't know the intentions of the engineer, you perhaps find the special part that is responsible for the 'wrong' results. You can 'fix' the machine by replacing that part (assuming it's an engineering mistake), or you assume it doesn't calculate the sum of two numbers and try to find out what this Foobar-value might be for.

But we can avoid this problem by knowing that theoretical devices work on different domains than physical devices. And this is, what technology/engineering is all about: Find a mapping between both, that is reasonable under real-world constraints.

"So reductionism is wrong - a thing can be more than the sum of it's parts (since "thing" includes action)."

The problem with this statement is that you don't define what you mean by sum. I for one cannot imagine what the term 'fingers + palm + thumb' is supposed to mean. Apparently by sum you don't mean arithmetic sum, but something different. Perhaps by 'sum' you mean something like 'put those ingredients into a beaker, shake it a little and then see what you get'.

And of course, if you defined 'sum', you'll need to define 'more' (and 'less') in this context. Perhaps you'll see that it's about the language and how we use it. We overload many words to mean different things and too often we use the special meaning of a word in a context where it doesn't belong.

"To be clear then, your objection is that any physical device that seems to add is doing something different from a "theoretical" device that doesn't actually get built."

As long as the domain on which they act is different, they are doing different things. If your theoretical device includes the whole universe as the domain on which it acts, then you generally cannot prove that they are doing different things (Halting problem).

But I do not say that you cannot create an AI that seems to act like a human, and I am not saying that it wouldn't be thinking like a human or that it wouldn't be conscious.

"Lukas, I don't understand your objection at all. How does disrupting the physical adding machine mid-process prove that it isn't doing addition? One can also disrupt electronic computers mid-process..."

I didn't say it doesn't do addition; I said that it doesn't do the same addition that the 'theoretical' adder is doing. That's what Elizier called 'artificial addition'.

All you can say is that the physical adder will most of the time do a 'physical addition' that corresponds to the 'theoretical addition'; but you need to make a lot of assumptions about the environment of the physical adder (it doesn't melt, it doesn't explode etc.), and those assumptions don't need to hold.

You don't need to make assumptions for the theoretical adder: You define it to do addition.

"Here's a question from a layman: if untold trillions of new universes are being created all the time, where is all that energy coming from to create them?"

Well, you've got the same problem with a single world: Where did the energy for our 'single' Universe when 'it was created' came from?

The problem here is that you assume that universes are created which did not exist before; in this case you indeed need to take the energy from somewhere. But as I understand, they never did not exist (beware of double negation!). They already existed before the split took place in your personal memory.

But somehow I still can't buy into this thing; where is the symmetry? Why do splits happen into the future, but not into the past?

Of course, we evaluate the past according to the information we retrieve over time (that's the whole point of Bayes/Markov, isn't it?). In this way you can say, that with every bit of information/evidence, our memory makes a split into the past. In this way 'fresher' memory gets mixed up with 'decaying' memory and thus we get a different/more diffuse image of the past.

But it doesn't sound the same like the 'future' splits. We don't have a fresh memory of the future; taking the example of lotteries. We don't remember their outcome seconds before.