This post is meant for readers familiar with algebra and derivatives, but want to deepen their understanding and/or need a refresh.

Linear functions

Let's start with a family of very basic functions: the linear functions, expressed as $f\left(x\right)=ax+b$. You might remember its derivative is $f^{\prime }\left(x\right)=a$, because $x$ is multiplied by $a$ and the constant $b$ "disappears" when taking the derivative. This is correct, but let's actually calculate the derivative. Since $f\left(x\right)$ is a linear function, $f^{\prime }\left(x\right)$ is the same for all $x.$ That is, a linear function "goes up" with the same "speed" everywhere, as can be seen in the following graph for $g\left(x\right)=2x+5$:

For example, between $x=0$ and $x=1$, $f\left(x\right)$ increases with $2$, just like it does between e.g. $x=2$ and $x=3$. Therefore, determining the average slope between $x$ and $x+d$ will do. The average slope between $x$ and $x+d$ is how much $f\left(x\right)$ increases between $x$ and $x+d$, divided by the difference  between $x$ and $x+d$ (which is $d$). Let $d=1$, in which case we don't have to do the division, as $f^{\prime }\left(x\right)=\frac{f(x+1)-f\left(x\right)}{1}=f(x+1)-f\left(x\right)$. Filling in $ax+b$ for $f\left(x\right)$ and $a(x+1)+b$ for $f(x+1)$, we get:

$f^{\prime }\left(x\right)=f(x+1)-f\left(x\right)=\left(a\right(x+1)+b)-(ax+b)=ax+a+b-ax-b=a$

There it is! $f^{\prime }\left(x\right)=a$. So for $g\left(x\right)=2x+5$, where $a=2$, this means $g^{\prime }\left(x\right)=2$. 

Polynomials (and more)

Polynomials are functions with the following form:

$f\left(x\right)=a_0{x}^n+a_1{x}^{n-1}+...+a_{n-1}x+a_n$

Determining their derivative is a bit more tricky than determining the derivative of a linear function, because now, the derivative isn't necessarily the same everywhere. After all, take $g\left(x\right)=x^2$:

We can see this is a curved line, and so the derivative is constantly changing. We can sill do something like the "trick" we did with linear functions, but we can't determine $f^{\prime }\left(x\right)$by looking how $f\left(x\right)$ changes between $x$ and $x+1$: that would assume $f^{\prime }\left(x\right)$ is the same between $x$ and $x+1$, which isn't true. For $x$ and $x+0.001$, we would have a better estimate of $f^{\prime }\left(x\right)$, but we'd still assume $f^{\prime }\left(x\right)$ to be constant between these values. We need to determine how $f\left(x\right)$ changes between $x$ and some $x+d$, where $d$ needs to approach zero: the smaller $d$ gets, the more accurate our calculation for $f^{\prime }\left(x\right)$ becomes. We can do this using limits:

$f^{\prime }\left(x\right)={lim}_{d\to 0}\frac{f(x+d)-f\left(x\right)}{d}$

(Since $d$ isn't 1 now, we need to do the division.) We can read this as follows: what value does $\frac{f(x+d)+f\left(x\right)}{d}$ approach when $d$ approaches $0$?

Let's do this for the simple polynomial $g\left(x\right)=x^2$:

$g^{\prime }\left(x\right)={lim}_{d\to 0}\frac{g(x+d)-g\left(x\right)}{d}={lim}_{d\to 0}\frac{(x+d{)}^2-x^2}{d}={lim}_{d\to 0}\frac{x^2+2dx+d^2-x^2}{d}={lim}_{d\to 0}\frac{2dx+d^2}{d}$

$g^{\prime }\left(x\right)={lim}_{d\to 0}(\frac{2dx}{d}+\frac{d^2}{d})={lim}_{d\to 0}(2x+d)$

When $d$ approaches $0$, $2x+d$ becomes $2x$:

$g^{\prime }\left(x\right)={lim}_{d\to 0}(2x+d)=2x$.

So for $g\left(x\right)=x^2$, $g^{\prime }\left(x\right)=2x$. You might have learned the general rule:

For $f\left(x\right)=a{x}^b$, $f^{\prime }\left(x\right)=ab{x}^{b-1}$

This is known as the power rule, and indeed works for $g\left(x\right)=x^2$, where $a=1$ and $b=2$and $1\ast 2\ast x^{2-1}=2x^1=2x$. It also works for the linear function $h\left(x\right)=2x$, where $a=2$ and $b=1$: $h^{\prime }\left(x\right)=1\ast 2x^{1-1}=1\ast 2x^0=1\ast 2\ast 1=2$. But does it work in general? Yes, and we can proof it. Let's first proof it works for all natural numbers ($0,1,2,3,...$) $b$:  $b\in N$. We need the product rule and mathematical induction for this proof though, so let's discuss those first.

Product rule

The product rule states that when $f\left(x\right)=g\left(x\right)\ast h\left(x\right)$, $f^{\prime }\left(x\right)=g^{\prime }\left(x\right)\ast h\left(x\right)+g\left(x\right)\ast h^{\prime }\left(x\right)$. So when e.g. $g\left(x\right)=2x$ and $h\left(x\right)=3x^2$, $f\left(x\right)=2x\ast 3x^2$ and

$f^{\prime }\left(x\right)=g^{\prime }\left(x\right)\ast h\left(x\right)+g\left(x\right)\ast h^{\prime }\left(x\right)=2\ast 3x^2+2x\ast 6x=6x^2+12x^2=18x^2$. We can show the product rule is correct by determining what $f^{\prime }\left(x\right)$ should be using the original definition of the derivative:

$f^{\prime }\left(x\right)={lim}_{d\to 0}\frac{f(x+d)-f\left(x\right)}{d}=\frac{g(x+d)\ast h(x+d)-g\left(x\right)\ast h\left(x\right)}{d}$

Since we want to write $f^{\prime }\left(x\right)$ as $g^{\prime }\left(x\right)\ast h\left(x\right)+g\left(x\right)\ast h^{\prime }\left(x\right)$, let's rewrite the divisor to include the terms $g(x+d)-g\left(x\right)$ and $h(x+d)-h\left(x\right)$:

$f^{\prime }\left(x\right)={lim}_{d\to 0}\frac{h\left(x\right)\left(g\right(x+d)-g(x\left)\right)+g(x+d)\left(h\right(x+d)-h(x\left)\right)}{d}$

and note that indeed, $h\left(x\right)\left(g\right(x+d)-g(x\left)\right)+g(x+d)\left(h\right(x+d)-h(x\left)\right)=$

$h\left(x\right)g(x+d)-g\left(x\right)h\left(x\right)+g(x+d)h(x+d)-h\left(x\right)g(x+d)=$

$g(x+d)h(x+d)-g\left(x\right)h\left(x\right)$, which was our original divisor.

Simplifying $f^{\prime }\left(x\right)={lim}_{d\to 0}\frac{h\left(x\right)\left(g\right(x+d)-g(x\left)\right)+g(x+d)\left(h\right(x+d)-h(x\left)\right)}{d}$, we get

$f^{\prime }\left(x\right)={lim}_{d\to 0}\frac{h\left(x\right)\left(g\right(x+d)-g(x\left)\right)}{d}+{lim}_{d\to 0}\frac{g(x+d)\left(h\right(x+d)-h(x\left)\right)}{d}$

Since $h\left(x\right)$ doesn't contain $d$, we can take it outside the first limit term. We can also rewrite the second term:

$f^{\prime }\left(x\right)=h\left(x\right)\ast {lim}_{d\to 0}\frac{g(x+d)-g\left(x\right)}{d}+{lim}_{d\to 0}g(x+d)\ast {lim}_{d\to 0}\frac{h(x+d)-h\left(x\right)}{d}$

When $d$ approaches $0$, $li{m}_{d\to 0}g(x+d)$ becomes $g\left(x\right)$. Furthermore, by definition,

${lim}_{d\to 0}\frac{g(x+d)-g\left(x\right)}{d}=g^{\prime }\left(x\right)$ and ${lim}_{d\to 0}\frac{h(x+d)-h\left(x\right)}{d}=h^{\prime }\left(x\right)$,

so we now have $f^{\prime }\left(x\right)=h\left(x\right)\ast g^{\prime }\left(x\right)+g\left(x\right)\ast h^{\prime }\left(x\right)=g^{\prime }\left(x\right)\ast h\left(x\right)+g\left(x\right)\ast h^{\prime }\left(x\right)$,

which is the product rule!

Mathematical induction

Mathematical induction is a method for proving something is true for all natural numbers $N$. For example, say we want to proof that for every natural number $n\in N$, $S\left(n\right)=\frac{n(n+1)}{2}$, where $S\left(n\right)$ is simply $0+1+2+...+n$. We can do this by first showing the condition holds for $n=0$. That's Step 1, and yes, it does: $\frac{0(0+1)}{2}=0=S\left(0\right)$. Then, we show that if the condition holds for some $n$, it also holds for $n+1$. That's Step 2. So for this step we assume $S\left(n\right)=\frac{n(n+1)}{2}$, and need to show that $S(n+1)=\frac{(n+1)\left(\right(n+1)+1)}{2}$. That holds as well: if $S\left(n\right)=\frac{n(n+1)}{2}$, then $S(n+1)=0+1+2+...+n+(n+1)=S\left(n\right)+n+1$. Since for this step we assumed $S\left(n\right)=\frac{n(n+1)}{2}$, we have $S(n+1)=\frac{n(n+1)}{2}+n+1=\frac{n(n+1)}{2}+\frac{2(n+1)}{2}$. So $S(n+1)=\frac{n(n+1)+2(n+1)}{2}=\frac{(n+1)(n+2)}{2}=\frac{(n+1)\left(\right(n+1)+1)}{2}$.

So we now know that our condition holds for $n=0$ and that if it holds for some $n$, it must also hold for $n+1$. But then it holds for all natural numbers! Does our condition hold for $n=3$? Yes! It holds for $n=0$ by Step 1, so it holds for $n=1$ by Step 2; but then, since it holds for $n=1$, it also holds for $n=2$, again by Step 2. Applying Step 2 one more time gives that the condition $S\left(n\right)=\frac{n(n+1)}{2}$ holds for $n=3$ as well. And we can apply this process to every natural number!

Proof of the power rule for natural numbers

Using the product rule and mathematical induction, we can show that the power rule (for $f\left(x\right)=a{x}^b$, $f^{\prime }\left(x\right)=ab{x}^{b-1}$) works for all $b\in N$.

Step 1 is to show this is true for $b=0$. Yes: then $f\left(x\right)=a{x}^b=a{x}^0=a$, and $ab{x}^{b-1}=a\ast 0x^{0-1}=0=f^{\prime }\left(x\right)$. (Since $f\left(x\right)$ is constant ($a$), its derivative $f^{\prime }\left(x\right)$ is indeed $0$.

Step 2 is to show that if $f^{\prime }\left(x\right)=ab{x}^{b-1}$ for some $b\in N$ and $f\left(x\right)=a{x}^b$, then for $b+1$ and $g\left(x\right)=a{x}^{b+1}$, $g^{\prime }\left(x\right)=a(b+1)x^b$.

We can write $g\left(x\right)=a{x}^{b+1}$ as $g\left(x\right)=x\ast a{x}^b$. Then, define $h\left(x\right)=x$. Then $g\left(x\right)=h\left(x\right)\ast f\left(x\right)$, and then the product rule says $g^{\prime }\left(x\right)=h^{\prime }\left(x\right)\ast f\left(x\right)+h\left(x\right)\ast f^{\prime }\left(x\right)$. But by the assumption of Step 2, $f^{\prime }\left(x\right)=ab{x}^{b-1}$. Furthermore, $h^{\prime }\left(x\right)=1$. So $g^{\prime }\left(x\right)=1\ast a{x}^b+x\ast ab{x}^{b-1}=a{x}^b+abx{x}^{b-1}=a{x}^b+ab{x}^b=a(b+1)x^b$, which is what we wanted to proof!

So we have shown the power rule works for $b\in N$. We could extend this proof to e.g. cover negative integers for $b$ as well. But I'd like to use a different method of proof, that proofs the power rule works for $b\in R$. For this, we first need to know the chain rule, the constant multiple rule, Euler's number and how to take the derivative of the natural logarithm.

Chain rule

Define $f\left(x\right)=(3x{)}^2$. (Note this is distinct from $3x^2$.) We want to determine its derivative. We could say $f\left(x\right)=(3x{)}^2=9x^2$, which would make $f^{\prime }\left(x\right)=18x$. This is true, but let's take the opportunity to study the chain rule. Define $g\left(x\right)=3x$ and $h\left(x\right)=x^2$. We can then write $f\left(x\right)$ as $h\left(g\right(x\left)\right)$. Then:

$f^{\prime }\left(x\right)={lim}_{d\to 0}\frac{f(x+d)-f\left(x\right)}{d}={lim}_{d\to 0}\frac{h\left(g\right(x+d\left)\right)-h\left(g\right(x\left)\right)}{d}$

Multiplying by $\frac{g(x+d)-g\left(x\right)}{g(x+d)-g\left(x\right)}$, which equals 1 and is allowed if $g(x+d)\ne g\left(x\right)$ (otherwise we are dividing by 0), gives:

$f^{\prime }\left(x\right)={lim}_{d\to 0}\frac{\left(h\right(g(x+d))-h(g\left(x\right)\left)\right)\ast \left(g\right(x+d)-g(x\left)\right)}{\left(g\right(x+d)-g(x\left)\right)\ast d}$

or $f^{\prime }\left(x\right)={lim}_{d\to 0}\frac{h\left(g\right(x+d\left)\right)-h\left(g\right(x\left)\right)}{g(x+d)-g\left(x\right)}\ast {lim}_{d\to 0}\frac{g(x+d)-g\left(x\right)}{d}$

Note that ${lim}_{d\to 0}\frac{h\left(g\right(x+d\left)\right)-h\left(g\right(x\left)\right)}{g(x+d)-g\left(x\right)}=h^{\prime }\left(g\right(x\left)\right)$, and ${lim}_{d\to 0}\frac{g(x+d)-g\left(x\right)}{d}=g^{\prime }\left(x\right)$. So we now have $f^{\prime }\left(x\right)=h^{\prime }\left(g\right(x\left)\right)\ast g^{\prime }\left(x\right)$. That's the chain rule, and it holds whenever we can write a function $f\left(x\right)$ as $f\left(x\right)=h\left(g\right(x\left)\right)$. Originally, we said $f\left(x\right)=(3x{)}^2$, with $g\left(x\right)=3x$ and $h\left(x\right)=x^2$. Then $h^{\prime }\left(g\right(x\left)\right)=2g\left(x\right)=2\ast 3x=6x$ and $g^{\prime }\left(x\right)=3$. According to the chain rule, then, $f^{\prime }\left(x\right)=6x\ast 3=18x$, which is also what we got by applying the power rule to $f\left(x\right)=(3x{)}^2=9x^2$.

Before, we temporarily assumed $g(x+d)\ne g\left(x\right)$. What if $g(x+d)=g\left(x\right)$? Well, then $h\left(g\right(x+d\left)\right)=h\left(g\right(x\left)\right)$, and $h\left(g\right(x+d\left)\right)-h\left(g\right(x\left)\right)=0$. Then $f^{\prime }\left(x\right)={lim}_{d\to 0}\frac{0}{d}=0$, and $g^{\prime }\left(x\right)={lim}_{d\to 0}\frac{g(x+d)-g\left(x\right)}{d}={lim}_{d\to 0}\frac{0}{d}=0$. So the chain rule would still apply, as $h^{\prime }\left(g\right(x\left)\right)\ast g^{\prime }\left(x\right)=h^{\prime }\left(g\right(x\left)\right)\ast 0=0=f^{\prime }\left(x\right)$.

Constant multiple rule

If $f\left(x\right)=a\ast g\left(x\right)$, $f^{\prime }\left(x\right)=a\ast g^{\prime }\left(x\right)$. This might make intuitive sense, but it also follows from the chain rule: define $h\left(x\right)=ax$ and $f\left(x\right)=h\left(g\right(x\left)\right)$. Then $f^{\prime }\left(x\right)=h^{\prime }\left(g\right(x\left)\right)\ast g^{\prime }\left(x\right)=a\ast g^{\prime }\left(x\right)$, which is the constant multiple rule. Indeed, this same rule also follows from the product rule: if $f\left(x\right)=a\ast g\left(x\right)$, define $h\left(x\right)=a$. Then $f\left(x\right)=h\left(x\right)\ast g\left(x\right)$ and $f^{\prime }\left(x\right)=h^{\prime }\left(x\right)\ast g\left(x\right)+h\left(x\right)\ast g^{\prime }\left(x\right)=0\ast g\left(x\right)+a\ast g^{\prime }\left(x\right)=a\ast g^{\prime }\left(x\right)$.

Euler's number and the natural logarithm

You might know that Euler's number $e$, which is chosen so that if $f\left(x\right)=e^x$, $f^{\prime }\left(x\right)=e^x$. You may also remember the natural logarithm $\ln$, where $e^{\ln x}=x$. What's the derivative of $f\left(x\right)=\ln x$? We can find it with the chain rule! Define $g\left(x\right)=e^{f\left(x\right)}=e^{\ln x}$ and $h\left(x\right)=e^x$. Then $g\left(x\right)=h\left(f\right(x\left)\right)$, and applying the chain rule gives $g^{\prime }\left(x\right)=h^{\prime }\left(f\right(x\left)\right)\ast f^{\prime }\left(x\right)=e^{\ln x}\ast f^{\prime }\left(x\right)$. But also, $g\left(x\right)=e^{\ln x}=x$, so $g^{\prime }\left(x\right)=1$. So we learn $g^{\prime }\left(x\right)=e^{\ln x}\ast f^{\prime }\left(x\right)=1$, or $x\ast f^{\prime }\left(x\right)=1$, and so $f^{\prime }\left(x\right)=\frac{1}{x}$. So for $f\left(x\right)=\ln x$, $f^{\prime }\left(x\right)=\frac{1}{x}$.

General proof of the power rule

Now we're ready to proof the power rule (for $f\left(x\right)=a{x}^b$, $f^{\prime }\left(x\right)=ab{x}^{b-1}$) works for $b\in R$. Let's rewrite $x$ as $e^{\ln x}$. Then $f\left(x\right)=a(e^{\ln x}{)}^b=a{e}^{b\ln x}$. Define $g\left(x\right)=a{e}^x$ and $h\left(x\right)=b\ln x$. Then $f\left(x\right)=g\left(h\right(x\left)\right)$, and via the chain rule (and the constant multiple rule) $f^{\prime }\left(x\right)=g^{\prime }\left(h\right(x\left)\right)\ast h^{\prime }\left(x\right)=a{e}^{b\ln x}\ast b\ast \frac{1}{x}=a{e}^{b\ln x}\ast \frac{b}{x}$. Remember $a{e}^{b\ln x}=a(e^{\ln x}{)}^b=a{x}^b$, so $f^{\prime }\left(x\right)=a{x}^b\ast \frac{b}{x}=\frac{ab{x}^b}{x}=ab{x}^{b-1}$, which is what we need to proof the power rule for $b\in R$.

Local maxima, local minima and second derivatives

As you might know, polynomials like $f\left(x\right)=-x^2$  can have local maxima (or peaks, where the graph first goes up and then goes down) and local minima (or minima, where the graph first goes down and then goes up). When a graph goes up, the derivative is positive; when it goes down, the derivative is negative. In the peak, the derivative must be $0$! It's similar for valleys - the derivative is $0$there, too. That means we can find local maxima and local minima by setting the derivative to $0$! For $f\left(x\right)=-x^2$, $f^{\prime }\left(x\right)=-2x$. $f^{\prime }\left(x\right)=0$ gives $-2x=0$ and thus $x=0$. Therefore, there must be a local maximum or minimum at $f\left(0\right)$. Which is it? Well, note that in a local maximum, the derivative must be decreasing (through $0$): otherwise, the graph wouldn't first go up and then go down. But if the derivative is decreasing, the derivative of the derivative, called the second derivative (written $f^{\prime \prime }\left(x\right)$), must be negative! Conversely, in a local minimum, the second derivative must be positive. For $f\left(x\right)=-x^2$, $f^{\prime }\left(x\right)=-2x$ and $f^{\prime \prime }\left(x\right)=-2<0$. So $f\left(x\right)=0$ is a local maximum!

Now consider $g\left(x\right)=x^3+x^2$. We have $g^{\prime }\left(x\right)=3x^2+2x$, and $g^{\prime }\left(x\right)=0$ gives $3x^2+2x=0$ or $x(3x+2)=0$. Then $x=0\vee 3x+2=0$ and so $x=0\vee x=-\frac{2}{3}$. We have a local maximum or minimum in $x=0$ and a local maximum or minimum in $x=-\frac{2}{3}$. $g^{\prime \prime }\left(x\right)=6x+2$, $g^{\prime \prime }\left(0\right)=6\ast 0+2=2>0$ and $g^{\prime \prime }(-\frac{2}{3})=6\ast -\frac{2}{3}+2=-2<0$. Therefore, we have a local minimum in $x=0$ and a local maximum in $x=-\frac{2}{3}$.

Multivariable functions

Multivariable functions are functions with, well, more than one variable. Take for example $f(x,y)=2x+3y$. For $x=4$ and $y=5$, we have $f(4,5)=2\ast 4+3\ast 5=23$. Or we can take $g(x,y,z)=x+5y+2z$, with $g(1,1,1)=1+5+2=8$.

Partial derivatives

A partial derivative of a multivariable function is determined by treating all but one variable like constants and taking the derivative with respect to the one variable left. For example, for $f(x,y)=2x+3y$, we can derive with respect to $x$: $f_x^{\prime }(x,y)=2$ and with respect to $y$: $f_y^{\prime }=3$. More generally, for any 2-variable function $f(x,y)$, $f_x^{\prime }(x,y)={lim}_{d\to 0}\frac{f(x+d,y)-f(x,y)}{d}$ and $f_y^{\prime }(x,y)={lim}_{d\to 0}\frac{f(x,y+d)-f(x,y)}{d}$. For $f(x,y)=2x+3y$, this means $f_x^{\prime }(x,y)={lim}_{d\to 0}\frac{f(x+d,y)-f(x,y)}{d}={lim}_{d\to 0}\frac{\left(2\right(x+d)+3y)-(2x+3y)}{d}={lim}_{d\to 0}\frac{(2x+2d+3y)-2x-3y)}{d}$, which indeed simplifies to $f_x^{\prime }(x,y)={lim}_{d\to 0}\frac{2d}{d}={lim}_{d\to 0}2=2$.