Introduction

A recent popular tweet did a "math magic trick", and I want to explain why it works and use that as an excuse to talk about cool math (functional analysis). The tweet in question:

This is a cute magic trick, and like any good trick they nonchalantly gloss over the most important step. Did you spot it? Did you notice your confusion?

Here's the key question: Why did they switch from a differential equation to an integral equation? If you can use $(1-x{)}^{-1}=1+x+x^2+...$ when $x=\int$, why not use it when $x=d/dx$? 

Well, lets try it, writing $D$ for the derivative:

$\begin{array}{ccc}{f}^{\prime }& =& f\\ (1-D)f& =& 0\\ f& =& (1+D+D^2+...)0\\ f& =& 0+0+0+...\\ f& =& 0\end{array}$

So now you may be disappointed, but relieved: yes, this version fails, but at least it fails-safe, giving you the trivial solution, right?

But no, actually $(1-D{)}^{-1}=1+D+D^2+...$ can fail catastrophically, which we can see if we try a nonhomogeneous equation like $f^{\prime }=f+e^x$ (which you may recall has solution $x{e}^x$):

$\begin{array}{ccc}{f}^{\prime }& =& f+e^x\\ (1-D)f& =& e^x\\ f& =& (1+D+D^2+...)e^x\\ f& =& e^x+e^x+e^x+...\\ f& =& \infty ?\end{array}$

However, the integral version still works. To formalize the original approach: we define the function $I$ (for integral) to take in a function $f\left(x\right)$ and produce the function $If$ defined by $If\left(x\right)={\int }_0^{x}f\left(t\right)dt$. This rigorizes the original trick, elegantly incorporates the initial conditions of the differential equation, and fully generalizes to solving nonhomogeneous versions like $f^{\prime }=f+e^x$ (left as an exercise to the reader, of course).

So why does $(1-D{)}^{-1}=1+D+D^2+...$ fail, but $(1-I{)}^{-1}=1+I+I^2+...$ works robustly? The answer is functional analysis!

Functional Analysis

Savvy readers may already be screaming that the trick $(1-x{)}^{-1}=1+x+x^2+...$ for numbers only holds true for $|x|<1$, and this is indeed the key to explaining what happens with $D$ and $I$! But how can we define the "absolute value" of "the derivative function" or "the integral function"?

What we're looking for is a norm, a function that generalizes absolute values. A norm is a function $x\mapsto ||x||$ satisfying these properties:

1. $||x||\ge 0$ for all $x$ (positivity), and $||x||=0$ if and only if $x=0$ (positive-definite)
2. $||x+y||\le ||x||+||y||$ for all $x$ and $y$ (triangle inequality)
3. $||cx||=|c|\cdot ||x||$ for all $x$ and real numbers $c$, where $|c|$ denotes the usual absolute value (absolute homogeneity)

Here's an important example of a norm: fix some compact subset of $R$, say $X=[-10,10]$, and for a continuous function $f:X\to R$ define $||f|{|}_{\infty }={max}_{x\in X}|f\left(x\right)|$, which would commonly be called the $L^{\infty }$-norm of $f$. (We may use a maximum here due to the Extreme Value Theorem. In general you would use a supremum instead.) Again I shall leave it to the reader to check that this is a norm.

This example takes us halfway to our goal: we can now talk about the "absolute value" of a continuous function that takes in a real number and spits out a real number, but $D$ and $I$ take in functions and spit out functions (what we usually call an operator, so what we need is an operator norm).

Put another way, the $L^{\infty }$-norm is "the largest output of the function", and this will serve as the inspiration for our operator norm. Doing the minimal changes possible, we might try to define $||I||={max}_{f\text{continuous}}||If|{|}_{\infty }$. There are two problems with this:

1. First, since $I$ is linear, you can make $||If|{|}_{\infty }$ arbitrarily large by scaling $f$ by 10x, or 100x, etc. We can fix this by restricting the set of valid f for these purposes, just like how for the $L^{\infty }$ example restricted the inputs of $f$ to the compact set $X=[-10,10]$. Unsurprisingly nice choice of set to restrict to is the "unit ball" of functions, the set of functions with $||f|{|}_{\infty }\le 1$.   
2. Second, we must bid tearful farewell to the innocent childhood of maxima, and enter the liberating adulthood of suprema. This is necessary since $f$ ranges over the infinite-dimensional vector space of continuous functions, so the Heine-Borel theorem no longer guarantees the unit ball is compact, and therefore the extreme value theorem no longer guarantees that we will attain a maximum.

So the proper definition of the norm of $I$ and $D$ are: 

$||I||={sup}_{f\text{continuous},||f|{|}_{\infty }\le 1}||If|{|}_{\infty }$

$||D||={sup}_{f\text{continuous},||f|{|}_{\infty }\le 1}||Df|{|}_{\infty }$

(and you can define similar norms for any linear operator, including $I^2$, etc.) A good exercise is to show these equivalent definitions of the operator norm for any linear function L:

$\begin{array}{ccc}||L||& =& \underset{f\text{continuous},||f|{|}_{\infty }\le 1}{sup}||Lf|{|}_{\infty }\\ & =& \underset{f\text{continuous},||f|{|}_{\infty }=1}{sup}||Lf|{|}_{\infty }\\ & =& inf\{b\ge 0:||Lf|{|}_{\infty }\le b||f|{|}_{\infty }\text{for all}f\}\end{array}$

So another way of thinking of the operator norm is the maximum stretching factor of the linear operator. The third definition also motivates the terminology of bounded linear operators: each such $b$ is a bound on the operator $L$, and the least such bound is the norm. Fun exercise: show that a linear operator is bounded if and only if it is continuous (with respect to the correct topologies). Hint: you'll need to work in infinite dimensional spaces here, because any finite-dimensional linear operator must be bounded.

Now let's actually compute these norms! For $I$, remember that our $L^{\infty }$-norm is defined over the interval $X=[-10,10]$. First observe that for the constant function $f\left(x\right)=1$, $If\left(x\right)=x$, so $||If|{|}_{\infty }=10$. Thus $||I||\ge 10$. To show that this is indeed the maximum we use the triangle inequality for integrals:

$\begin{array}{ccc}||If||& =& \underset{x\in X}{max}|{\int }_0^{x}f\left(t\right)dt|\\ & \le & \underset{x\in X}{max}{\int }_0^x|f\left(t\right)|dt\\ & \le & \underset{x\in X}{max}{\int }_0^{x}1dt\\ & =& \underset{x\in X}{max}x=10\end{array}$

So we have shown $||I||=10$! Put a pin in that while we check $||D||$.

For $D$, we have a problem: for any positive number $k$, $D\left(e^{kx}\right)=k{e}^{kx}$. In other words, $D$ can stretch functions by any amount, so it has no norm, or we'd write $||D||=\infty$ (and I promise this is a failure of $D$, not of our definitions). Put another way, $D$ is not bounded as a linear operator, since it can stretch functions by an arbitrary amount. 

But now let's return to $I$. We said that $||I||=10$ (if we're defining it relative to the $L^{\infty }$-norm on $X=[-10,10]$), but isn't $(1-x{)}^{-1}=1+x+x^2+...$ only true when $|x|<1$? For real numbers, yes, but for operators, something magical happens: $||I^n||\ne ||I|{|}^n$! (Its like there's a whole algebra of these operators...) 

In fact, you can show that $||I^n||$ assumes its maximum value when applied to the constant function $f\left(x\right)=1$, and hence have $||I^n||=\frac{1}{n!}||I|{|}^n$. Since $n!$ grows faster than exponential functions, $||I^n||$ is converging to 0 quickly, so $1+I+I^2+I^3+...$ is a Cauchy sum, and it is then straightforward to show that the limit is the multiplicative inverse of $1-I$. Thus, $(1-I{)}^{-1}=1+I+I^2+...$ is a valid expression that you can apply to any continuous (or bounded) function $f$ on any compact set $X$. This convergence happens regardless of the choice of the compact set, though it will happen at different rates, analogous to uniform convergence on compact sets.

Summary 

• Writing $D$ for derivative and $I$ for integral, we showed that $(1-D{)}^{-1}=1+D+D^2...$ can fail, even though $(1-I{)}^{-1}=1+I+I^2...$ is always true.
• To explain this, we have to show that $I$ is fundamentally better behaved than $D$, in a way analogous to $|x|<1$.
• We built this up in two steps. First, we defined the $L^{\infty }$-norm for real-valued functions, which lets you say how "large" those functions are. Then, we extended this to function-valued functions (operators), having to make two slight modifications along the way.
• With this machinery in place, we could show that $||I||<\infty$, or we can say that $I$ is bounded. The resulting norm depends on the domain of the functions $f$ under consideration, but any compact domain is allowable. Also, since $||I^n||=\frac{1}{n!}||I|{|}^n$, the exact value doesn't matter since the norm of each term goes to 0.
• Since $||I^n||\to 0$ sufficiently quickly, we can say that $1+I+I^2+I^3+...$ is Cauchy as a sequence of operators. In other words, if you apply the partial sums $1,1+I,1+I+I^2,...$ as operators to any function $f$, the functions $f,f+If,f+If+I^{2}f,...$ will converge with respect to the $L^{\infty }$-norm. Writing $Lf$ for the function they converge to, it follows that $(1-I)Lf=f$, so we may write $L=1+I+I^2+I^3+...=(1-I{)}^{-1}$ as a statement about linear operators.
• In contrast, $D$ is unbounded as an operator, meaning $||D||=\infty$. Thus algebra tricks like $(1-D{)}^{-1}=1+D+D^2...$ will break down if you put in the wrong function $f$.