(This is the seventh post in a sequence on Machine Learning based on this book. Click here for part I.)

Meta

A more accurate title for this post would be "Meta, Error bounds, Test Data, Meta-Learning, Error decomposition, and an optional introduction to general probability spaces." The first item is me trying to categorize the stuff we've been doing in this sequence so far, which is meta-learning but nonetheless different from the Meta-Learning item, which is about a Machine Learning technique.

Categorizing Machine Learning Insights

Although the book doesn't do this, I think taking some time to reflect on how exactly we've been making progress is very worthwhile. I would categorize the material we've covered so far as follows:

• (1) Carving out relevant and learnable classes out of ML problem space
• (1.1) Defining such classes and establishing results about their limits
• Examples: Linear Predictors (post 4), Convex (bounded, Lipschitz / smooth) problems (post 5), the negative result on convex problems (post 5)
• (1.2) Finding algorithms that learn these classes and prove that they work.
• Examples: Perceptron (post 4), Linear Programming (post 4), applied linear algebra (post 4), Stochastic Gradient Descent (post 6)
• (2) Extending the usability of such classes
• Examples: Primarily surrogate loss functions (post 5); also the mapping of inhomogeneous linear problems to linear problems (post 4), even if it's just a detail. There will be more stuff here in future posts.
• (3) General (as in, widely applicable) theoretical work
• Examples: The general learning model, work on overfitting, the PAC learning framework, the basic error decomposition, the ERM approach (all post 1), the uniform convergence property, the No-Free-Lunch Theorem, the VC dimension, the fundamental theorem of statistical learning (all post 2), the nonuniform learning framework, the SRM approach, the minimal description length, the basics of computational complexity (all post 3), meta-learning, error bounds via test data, and advanced error decomposition (later in this post)

Not quite covered: coming up with learning algorithms. So far, that's just Stochastic Gradient Descent, which I have listed as a solution to solve convex Lipschitz/smooth bounded problems, but it's not restricted to that. Also, neural networks supposedly have an impressive track record of fitting all sorts of functions, and I don't expect there to be analogous theoretical results.

Based on this categorization (which is not perfect – much of (3) is only about binary classification), the entire first part of the book which was covered in posts I-III is only about broad theoretical work, hence "foundations". The remaining book jumps around between all three. Note that my adaption isn't always linear.

Error Bounds

One way to motivate this chapter is that we wish to find better guarantees for the quality of predictors. We usually either care about the term $\ell \left(A_{\text{ERM}}\right(S\left)\right)-\ell \left(h^{\ast }\right)$ (where $h^{\ast }$ is the predictor with minimal real error in $H$) or about ${max}_{h\in H}\left[\ell \right(h)-{\ell }_S(h\left)\right]$. It turns out that both are closely related. We'll call either of these the "gap". Now one can alternatively ask

• "given that I have $m$ data points, what is the smallest upper-bound on the gap that I can prove?"; or
• "given that I want to establish an upper bound of $ϵ$ on the gap, how many data points do I need?"

where, in both cases, the guarantee will hold with some probability $1-\delta$, rather than with certainty. The first would be an error bound, the second a sample complexity bound, but they're equivalent: given an error bound, one can take the corresponding equation, which will have the form $\ell \left(A_{\text{ERM}}\right(S\left)\right)-\ell \left(h^{\ast }\right)\le ⟨\text{some term which depends on}m⟩$, set the left part to $ϵ$ and solve it for $m$, which will yield a sample complexity bound. Going into the opposite direction is no harder.

Musings on past approaches

The punchline of this section is that all bounds we have looked at so far are based on the performance of the classifier on the data it's been trained on. Given that the output predictor may be highly dependent on patterns that only exist in the training data but not in the real world, this is difficult – and, as we know from the No-Free-Lunch Theorem, even impossible without making further assumptions.

This implies that we must have made such assumptions, and indeed we have. In addition to the empirical error, our current bounds have been based on

• (1) the hypothesis class
• (2) the properties of the classifier that the learner guarantees.

By (2), I mean that the classifier is in the set ${\text{argmin}}_{h\in H}\left[{\ell }_S\right(h\left)\right]$ in the case of $A_{\text{ERM}}$, in the class ${\text{argmin}}_{h\in H}\left[{\ell }_S\right(h)+{ϵ}_n(\delta ,h\left)\right]$ in the case of $A_{\text{SRM}}$; and in the case of $A_{\text{SGD}}$ we've used the update rule in the proof that derived the error bound.

Deriving error bounds with test data

By testing the classifier on data that the learner didn't have access to, we can toss out (1) and (2) in favor of a purely statistical argument, which results in a bound that is more general, much easier to prove, and stronger. The downside is that it requires additional data, whereas the previous bounds were "free" in that regard.

To be more precise, the idea now is that we split our existing data into two sequences $S$ and $T$; we train a learner on $S$ and we test the output predictor $A\left(S\right)$ on $T$ to obtain an estimate of the real error. In symbols, we use ${\ell }_T\left(A\right(S\left)\right)$ as our estimate for $\ell \left(A\right(S\left)\right)$, where ${\ell }_T$ is defined just like ${\ell }_S$ only, of course, using $T$ instead of $S$ as the set (this definition is implicit if one simply regards the definition of ${\ell }_S$ as applying to $\ell$ with any set $S$ as a subscript). So ${\ell }_T\left(h\right)=\frac{1}{|T|}|\left\{\right(x,y)\in T|h(x)\ne y\}|$. We will call the output of ${\ell }_T$ the test error.

In general, the real error does not equal the test error (in symbols, $\ell \left(A\right(S\left)\right)\ne {\ell }_T\left(A\right(S\left)\right)$) because the test error is based on the test sequence $T$ which may be unrepresentative. However, the real error equals the expected test error (we'll prove this in the optional chapter at the end of the post). Therefore, we only need an upper-bound on the difference between the expected value and the actual value of random variables (this is the aforementioned statistical argument), and a theorem to that end has already been established by relevant literature: Hoeffding's Inequality.

Let ${\theta }_1,...,{\theta }_m$ be i.i.d. RVs that range over the interval $[0,1]$. Then, for all $ϵ\in R_{+}$, it holds that $Pr(|\overline{\theta }-E[\overline{\theta }]|>ϵ)\le 2e^{-2{ϵ}^{2}m}$, where $\overline{\theta }:=\frac{1}{m}{\sum }_{i=1}^m{\theta }_i$.

This uses the same notational shortcut that is used all the time for Random Variables (RVs), namely pretending as if RVs are numbers when in reality they are functions. Importantly, note that $\overline{\theta }$ depends on the input randomness, whereas $E[\overline{\theta }]$ is a constant.

In our case, the random variables we are concerned with are the point-based loss functions applied to our predictor for our instances in $T$, i.e., if $T=\left(\right(x_1,y_1),...,(x_t,y_t\left)\right)$, then our random variables are ${\theta }_1={\ell }_{(x_1,y_1)}\left(h\right),...,{\theta }_t={\ell }_{(x_t,y_t)}\left(h\right)$. They are random variables if we regard the points they're based on as unknown (otherwise they'd just be numbers). Their mean, $\overline{\theta }=\frac{1}{|T|}{\sum }_{i=1}^{|T|}{\ell }_{(x_i,y_i)}\left(h\right)$ equals the test error, ${\ell }_T\left(h\right)$, whose randomness ranges over the choice of all of $T$. So $\overline{\theta }={\ell }_T\left(h\right)$ and $E\left(\overline{\theta }\right)=\ell \left(h\right)$. Given this, Hoeffding's inequality tells us that $Pr\left(|\ell \right(h)-{\ell }_T(h)|>ϵ)\le 2e^{-2{ϵ}^{2}m}$, provided that all loss functions range over $[0,1]$ (if not, then as long as they are still bounded, a generalized version of this theorem exists). Furthermore, if one drops the $||$, the failure probability halves, i.e. $Pr\left(\ell \right(h)-{\ell }_T(h)>ϵ)\le e^{-2{ϵ}^{2}m}$. Setting $\delta :=e^{-2{ϵ}^{2}m}$ and solving for $ϵ$, we obtain $ϵ=\sqrt{\frac{\ln \left(1/\delta \right)}{2m}}$. Thus, we have established the following:

If $T$ is a sequence of i.i.d examples sampled by the same distribution that generated $S$, but is disjoint from $S$, then the inequality ${\ell }_T\left(A\right(S\left)\right)-\ell \left(A\right(S\left)\right)\le \sqrt{\frac{\ln \left(1/\delta \right)}{2|T|}}$ holds with probability at least $1-\delta$ over the choice of $T$.

For a brief comparison: a bound provided by the (quantitative version of) the fundamental theorem of statistical learning states that $|{\ell }_S\left(h\right)-\ell \left(h\right)|\le \sqrt{C\frac{d+\ln \left(1/\delta \right)}{m}}$, provided that $H$ has VC-dimension $d\in N$. So this bound relies on a property of the hypothesis class (and if that property doesn't hold, it doesn't tell us anything), whereas our new bound always applies. And as a bonus, the new bound doesn't have the constant $C$ but instead has a nice 2 in the denominator.

Meta-Learning

The book now goes on to talk about how this idea can be used for model selection. Let's begin by summarizing the argument.

Model Selection ...

Suppose we have several possible models for our learning task, or alternatively one model with different parameters (maximal polynomial degree, step size of stochastic gradient descent, etc). Then we can proceed as follows: we split our training data into three sequences, the training sequence $S$, the validation sequence $V$, and the test sequence $T$. For each model we look at, we train the corresponding learner $A$ on $S$, then we test all output hypotheses $A_1\left(S\right),...,A_k\left(S\right)$ on $V$, i.e. we compute ${\ell }_V\left(A_1\right(S\left)\right),...,{\ell }_V\left(A_k\right(S\left)\right)$, and pick one with minimal validation error, i.e. we pick $A_j\left(S\right)$ such that $j\in {\text{argmin}}_{i\in \{1,...,k\}}{\ell }_V\left(A_i\right(S\left)\right)$. Finally, we compute ${\ell }_T\left(A_j\right(S\left)\right)$ and use that value to argue that the predictor performs well (but not to change it in any way).

Since the learners $A_1\left(S\right),...,A_k\left(S\right)$ haven't been given access to $V$, the validation error will be unbiased feedback as to which learner performs best, and since even that learner has been computed without using $T$, the test error is yet again unbiased feedback on the performance of our eventual pick $A_j\left(S\right)$. That is, provided that there are no spurious correlations across our data sets, i.e. provided that the i.i.d. assumption holds.

One can be even more clever, and trade some additional runtime for more training data. The test sequence $T$ remains as-is, but rather than partitioning the remaining data in $S$ and $V$ we partition it into a bunch of blocks of equal size, say $k$ blocks, and then, for each such block $i$, we

• train our learner A on all blocks $S_{\ne i}$
• compute the error on block $i$, i.e. ${\ell }_{S_i}\left(A\right(S_{\ne i}\left)\right)$

This way, we obtain $k$ different validation errors (one for each classifier which was trained on nine blocks and tested on the tenth); now we take the mean of all of them as our ultimate validation error. We do this entire thing for each learner $A$, take the one that performed best, and then we can even retrain that learner on all $k$ blocks to make sure we fully utilize all data we have.

Finally, just as before, we test that final predictor using the test data and output the result. This entire process is called $k$-fold cross validation.

... is just meta-learning

While this is all well and good, I think it's important to understand that the specifics are non-fundamental. What is novel here is the concept of obtaining unbiased feedback by computing the empirical error of a predictor based on a sequence that has not been used to learn that predictor. Beyond that, the observation is that it might be useful to do meta-learning, i.e. have several levels of learning, i.e. partition all possible hypotheses in a bunch of different classes, take the best from each class, and then take the best across all classes.

But the notion of exactly two levels is arbitrary. Suppose we have a bunch of models, and each model has some parameters. Then we can partition our training data in four sequences $S,V_1,V_2,T$. For each model, we choose a bunch of plausible parameter values, so say we have $n$ models and each one has $m$ parameter settings. For each such setting, we train the learner on $S$ and test it on $V_1$. We pick the optimal one among those $m$; this will be the predictor with the optimal parameters for our model. We do this for all $n$ models, leading to $n$ predictors, all of whom use the optimal parameters for their model. We test these $n$ predictors using $V_2$ and output the optimum yet again. Finally, we test that predictor on $T$.

Similarly, one could imagine four levels, or however many. The crucial thing is just that choosing the optimal predictor of the next lower level has to be done with data that hasn't been used to learn that predictor. So in the above example, for each [model and particular parameter setting], the respective output hypothesis just depends on $S$. If we then compute the optimal predictor of that model by looking at which of the $m$ predictors (corresponding to the $m$ parameter settings) performed best on $V_1$, this output predictor also depends on $V_1$. In fact, the procedure I just described is just one kind of learning algorithm that uses both $S$ and $V_1$ as training data. Similarly, if we take the predictor with the smallest error on $V_2$ across all $n$ models to obtain our final predictor, then this predictor depends on $S$ and $V_1$ and $V_2$ – in fact, the procedure I just described is just one kind of learning algorithm that uses $S$ and $V_1$ and $V_2$.

Differently put, to derive a predictor for a particular problem, one might choose to apply meta-learning, i.e. choose $k$ different models, train a predictor in each model, and then select the best of those $k$ predictors, based on their performance on training data which hasn't been used to train them – and each of those $k$ smaller problems is itself just another problem, so one might again choose to apply meta-learning, and so on.

The trick from the last section for saving data can also be used with more than 2 levels, although it quickly increases runtime.

The idea of using a test sequence is more fundamental (i.e. not just another level), since it's only used to obtain a certificate of the quality of the final predictor, but not for learning (in any way). As soon as one violates this rule (for example by choosing another predictor after the first one showed poor performance on the test data), the bound from Hoeffding's inequality is no longer applicable, since the test error is no longer an unbiased estimate of the real error.

Error Decomposition

So far, we've decomposed the error of a predictor like so:

$\ell \left(h\right)=\left(\ell \right(h)-\ell (h^{\ast }\left)\right)+\ell \left(h^{\ast }\right)$

where $h^{\ast }\in {\text{argmin}}_{h\in H}\left[\ell \right(h\left)\right]$. The term $\ell \left(h^{\ast }\right)$ is the approximation error, which is the lowest error achievable by any predictor in our hypothesis class ("how well does this class approximate the problem?") and the term $\left(\ell \right(h)-\ell (h^{\ast }\left)\right)$ is the estimation error ("how well did we estimate the best predictor?").

The usefulness of this approach is limited. One doesn't actually know the approximation error – if learning fails, it's not clear which of the two errors is at fault. Furthermore, both overfitting and underfitting is part of the estimation error, which suggests that it should be divided up further (whereas the approximation error can stay as-is). This is what we do now. Our new decomposition will rely heavily on having the kind of unbiased feedback from some independent data which we've been talking about in the previous chapter. Note, however, that while I'll only talk about the test sequence $T$ going forward, everything applies equally if one uses a validation sequence instead (i.e. training data which has not been given to the learner). In fact, using such data is the way to go if one intends to change the learner in any way upon seeing the various parts of the decomposition.

I find the notation with differences not very illustrative, which is why I'm making up a different one. The above decomposition can be alternatively written as

$\ell \left(h\right)\stackrel{}{\to }\ell \left(h^{\ast }\right)$

where we decompose the left term into the value of the arrow plus that of the right term. (Each error corresponds to adding "$+[\text{right term}-\text{left term}]$". If there are more than two terms, we decompose the leftmost term into the sum of all errors plus the rightmost term (actually, any term in the chain equals the sum of all errors to its right plus the rightmost term).

That said, here is a more sophisticated error decomposition:

The red terms are the ones we have access to. Note that it is not the case that each step is necessarily negative, so the values are not monotonically decreasing. Now let's look at this more closely.

"$\ell \left(h\right)⟶{\ell }_T\left(h\right)$" can be bounded using Hoeffding's inequality.

Let's pause already and reflect on that result. Our only terminal value is to maximize the leftmost term; the first arrow can be bounded tightly; and the second term is known. It follows that, if the test error is small we can stop right there (because the real error is probably also small). In fact, that was the point of the previous chapter. The remaining decomposition is only necessary if the test error is larger than what is acceptable.

"${\ell }_T\left(h\right)⟶{\ell }_S\left(h\right)$" is the difference between the unbiased and the biased error estimate – the error on the data the learner had no access to minus the error on the data it did have access to. It measures how much we overfit – which is quite useful given that this is another term we have access to. Thus, if the test error is large but the empirical error small, we might want to change our learner such that fits the training data less closely. However, we don't have any guarantee in this case that overfitting is the only problem. It's possible that we overfit and the approximation error is large.

If the test error and the empirical error are both large, that's when we're interested in the remaining part of this decomposition.

"${\ell }_S\left(h^{\ast }\right)⟶\ell \left(h^{\ast }\right)$" is usually negative, at least if we're using $A_{\text{ERM}}$ or something close to it.

"${\ell }_S\left(h^{\ast }\right)⟶\ell \left(h^{\ast }\right)$" can also be bounded by Hoeffding's inequality, because $h^{\ast }$ does not depend on $S$ (unless we've messed with the hypothesis class based on the training data).

Finally, $\ell \left(h^{\ast }\right)$ is the approximation error from our previous decomposition. It doesn't make sense to decompose this further – if the approximation error is large, our hypothesis class doesn't contain any good predictors and learning it is hopeless.

Thus, based on the previous 3 paragraphs, the empirical error is unlikely to be significantly larger than the total error of the output predictor, i.e. the approximation error. (It might be significantly smaller, though.) It follows that if the empirical error is high, then so is the approximation error (but not vice-versa), with the important caveat that one needs to be confident in the learner's ability to minimize empirical risk. If the problem is learning starcraft, then it's quite plausible that minimizing the empirical risk is so difficult that the empirical risk could be large even if the approximation error is very small.

To summarize...

... one can reason like this:

• Test error...
• is small $\to$ real error is probably small too, so everything's good
• is large $\to$ empirical error...
• is small $\to$ the learner probably suffers from overfitting${}^1$
• is large $\to$ the empirical error seems...
• easy to minimize $\to$ approximation error is probably large
• hard to minimize $\to$ ??? (the problem is just hard)

If the approximation error is large, then the problem is unrelated to our learner. In that case, the hypothesis class $H$ may be "too small"; perhaps it contains only linear predictors and the real function just isn't linear. Alternatively, it could also be the case that the real function is approximately linear, but we just didn't choose features that represent meaningful properties of the data points. For example, if the problem is learning a function that classifies emails as spam / not spam, one could imagine that someone was just really wrong about which features are indicative of spam emails. This is the earliest point of failure; in this case, learning might be impossible (or at least extremely difficult) even with a good hypothesis class and a good learner.

[1] While this is technically true, one might actually be fine with some amount of overfitting. If so, then this case isn't quite satisfactory, either; in particular, one doesn't know whether the approximation error is large or not. Another way to approach the question is to train the learner on parts of the training sequence first, say $S_1⊊\cdots ⊊S_k=S$, and examine the test error for each learner $A\left(S_j\right)$. If it goes down as we use more data, that's some evidence that the approximation error is low.

In general, this chapter is far from a full treatment of error decomposition.

Proof that E(test error) = real error

As mentioned, this chapter is completely optional (also note that the post up to this point is roughly as long as previous posts in this sequence). It's long bothered me that I didn't understand how a general probability space was defined, so if you share that frustration, this chapter will provide an answer. However, the connection to Machine Learning is very loose; one needs general probability spaces to prove the result, but the actual proof is quite easy, so this will be a lot of theory with a fairly brief application.

Preamble: General Probability Spaces

Before the general case is introduced, one is generally being taught two different important special cases of probability spaces. The first is the discrete one. Here, a probability space is given by a pair $(\Omega ,p)$, where $\Omega$ is a countable sample space and $p:\Omega \to [0,1]$ a function that assigns to each elementary event $\omega \in \Omega$ a probability. For any subset $A\subseteq \Omega$, the probability $P\left(A\right)$ is then simply defined as ${\sum }_{\omega \in A}p\left(\omega \right)$. And for a random variable $X:\Omega \to R$, we have the definition

(1): $E\left(X\right):={\int }_{-\infty }^{\infty }xf\left(x\right)\text{dx}$.

This is super intuitive, but it can't model how an event happens at a random point throughout a day. For that, we need the absolutely continuous case. This is where no one point has nonzero probability, but intervals do. In other words, one is in the absolutely continuous case whenever the probability distribution admits of a probability density function such that, for each interval, the probability of that interval is equal to the integral of that function over that interval. If $\Omega =R$ (covering only that case here), we can write the probability space as $(\Omega ,f)$ where for any interval $[a,c]\subseteq \Omega$, we have $P\left(\right[a,c\left]\right)={\int }_a^{c}f\left(x\right)\text{d}x$. This is also rather intuitive – one can just draw an analogy between probability mass and area. A single point has an "infinitely thin" area below it, so it has zero probability mass (and in fact the same is true for any countable collection of points because even the tiniest interval is uncountable), but an interval does have real area below it. The height of the area is then determined by $f$.

Given a random variable $X:\Omega \to R$, we have

(2): $E\left(X\right):={\int }_{-\infty }^{\infty }xf\left(x\right)\text{dx}$.

But neither of these is the general case, because one assumes that every point has nonzero probability and the other assumes that no point does. In the general case, a probability space looks like this:

$(\Omega ,\Sigma ,P)$

The first element $\Omega$ is the same as before – the sample space. But the $\Sigma$ is something new. It is called a $\sigma$-algebra, which is a set of subsets of $\Omega$, so $\Sigma \subseteq P\left(\Omega \right)$, and it consists of all those subsets which have a probability. It needs to fulfill a bunch of properties like being closed under complement and like $\Omega \in \Sigma$, but we don't need to concern ourselves with those.

The probability distribution is a function $P:\Sigma \to [0,1]$. So it's a function that assigns subsets of $\Omega$ a probability, but only those subsets that we declared to have a probability, i.e. only those subsets that are elements of our $\sigma$-algebra $\Sigma$. It must have a bunch of reasonable properties such as $P\left(\Omega \right)=1$ and $P\left(A\right)+P\left(B\right)=P(A\bigsqcup B)$, where the symbol $\bigsqcup$ is meant to indicate that $A$ and $B$ are disjoint.

( In the discrete case, we don't need to specify $\Sigma$ because we'd simply have $\Sigma =P\left(\Omega \right)$, i.e. every subset has a probability. I'm not sure how it would look in the continuous case, but maybe something like the set all of all subsets that can be written as a countable union of intervals plus a countable union of single points. )

The question relevant for us is now this: given a general probability space $(\Omega ,\Sigma ,P)$ and a random variable $X:\Omega \to P$, how do we compute the expected value of $X$? And the answer is this:

(3): $E\left(X\right):={\int }_{\Omega }X\left(\omega \right)\text{d}P\left(\omega \right)$

where $\int$ is the Lebesgue integral rather than the Riemann integral. But how does one compute a Lebesgue integral? Fortunately, we don't require the full answer to this question, because some of the easy special cases will suffice for our purposes.

Computing Lebesgue Integrals of Simple Functions

The setting to compute a Lebesgue integral is a bit more general than that of a general probability space, but since we're only trying to do the easy cases anyway, we'll assume our space has the same structure as before. However, we will rename our probability distribution $P$ into $\mu$ and call it a measure, so that our space now looks like this:

$(\Omega ,\Sigma ,\mu )$

And we'll also rename our random variable into $f$ to make it look more like a general function. So $f:\Omega \to R$ is the function whose Lebesgue integral ${\int }_{X}f\text{d}\mu$ we wish to compute.

Recall that $P$ assigns a probability to the subsets of $\Omega$ that appear in $\Sigma$, and that the entire space has probability 1. Analogously, $\mu$ assigns a measure to subsets of $\Omega$, and the entire space has measure 1. So let $Y$ be a subset that is "half" of $\Omega$, then $\mu \left(Y\right)=\frac{1}{2}$.

One now proceeds differently than with Riemann integrals. Rather than presenting the general case right away, we begin with the simplest kind of function, namely indicator functions. That is a function of the form $1_S$ for some $S\in \Sigma$, and it is simply 1 on $S$ and 0 everywhere else. And now, even though we don't understand how Lebesgue integrals work in general, in this particular case the answer is obvious:

${\int }_{\Omega }{1}_S\text{d}\mu :=\mu \left(S\right)$.

The function $1_S$ just says "I count $S$ once and I count nothing else", so the integral has to just be one times the size, i.e. the measure, of $S$. Which is $\mu \left(S\right)$.

So for example, ${\int }_{\Omega }{1}_Y\text{d}\mu =\mu \left(Y\right)=\frac{1}{2}$ and ${\int }_{\Omega }{1}_{\Omega }\text{d}\mu =\mu \left(\Omega \right)=1$.

Next, if $f$ is not itself an indicator function, but it can be written as a finite weighted sum of indicator functions, i.e. $f={\sum }_{i=1}^m{a}_i{1}_{S_i}$, then $f$ is called a simple function and we define

${\int }_{X}f\text{d}\mu :={\sum }_{i=1}^m{a}_i\mu \left(S_i\right)$

So for example, if $f=2\cdot 1_Y$, i.e. the function that is 2 on one half of the space and 0 on the other, then ${\int }_{X}f\text{d}\mu =2\mu \left(Y\right)=2\cdot \frac{1}{2}=1$.

And this where we stop because integrating simple functions is all we need.

An example

Let's return to our setting of general probability spaces. Here, the probability distribution becomes the measure. So let's say our space is $\Omega =[0,\frac{1}{2}]\cup \left\{10\right\}$. Our $\sigma$-algebra $\Sigma$ includes all sub-intervals of $[0,\frac{1}{2}]$ (closed or open or half-open), and all such intervals plus the point 10. Our probability distribution $P$ says that $P\left(\right\{10\left\}\right)=\frac{1}{2}$ and the probability mass of an interval in $[0,\frac{1}{2}]$ is just the size of the interval, so if $[a,c]\subseteq [0,\frac{1}{2}]$, then $P\left(\right[a,c\left]\right)=c-a$. Then, $P\left(\Omega \right)=1$. Recall that $P$ has to assign each set $M\in \Sigma$ a probability, which it now does.

Let's say $X:\Sigma \to R$ is a random variable given by $X=5\cdot 1_{[0,\frac{1}{3}]}+7\cdot 1_{\left\{10\right\}}$. Then, $X$ is a simple function – it is the weighted sum of two indicator functions – and we can compute the expected value of $X$ as

$E\left(X\right)={\int }_{\Omega }X\text{d}P=5\cdot P\left(\right[0,\frac{1}{3}\left]\right)+7\cdot P\left(\left\{10\right\}\right)=5\cdot \frac{1}{3}+7\cdot \frac{1}{2}=\frac{5}{3}+\frac{7}{2}=\frac{31}{6}$.

The actual proof

The real error of $h:=A\left(S\right)$ is

$\ell \left(h\right)=D\left(\right\{(x,y)\in X\times Y|h\left(x\right)\ne y\left\}\right)$

The test error ${\ell }_T\left(h\right)$ depends on the test sequence. Let $m$ be the length it the sequence, so that $T=\left(\right(x_1,y_1),...,(x_m,y_m\left)\right)$; then to compute the expected test error, we have to compute the Lebesgue integral

${\int }_{\Omega }{\ell }_T\left(h\right)\text{d}{D}^m$ where $\Omega =(X\times Y{)}^m$

So our measure for the space $\Omega$ is now $D^m$. We wish to write ${\ell }_T\left(h\right)$ as a sum of indicator functions. Recall that ${\ell }_T\left(h\right)=\frac{1}{m}|\left\{\right(x,y)\in T|h(x)\ne y\}|$, i.e. ${\ell }_T\left(h\right)$ counts the number examples which $h$ got wrong. Thus, by defining the sets $W_k:=\left\{\right((x_1,y_1),...,(x_m,y_m))\in \Omega |h(x_k)\ne y_k\}$ for all $k\in \{1,...,m\}$, we get the equality ${\ell }_T\left(h\right)={\sum }_{k=1}^m\frac{1}{m}\cdot 1_{W_k}$.

Thus, the integral simply equals the probability mass of the respective sets (multiplied by their respective weights). In symbols,

${\int }_{\Omega }{\ell }_T\left(h\right)\text{d}{D}^m={\sum }_{k=1}^m\frac{1}{m}{D}^m\left(W_k\right)$

The i.i.d. assumption tells us that $D^m\left(W_k\right)=D\left(\right\{(x,y)\in X|h\left(x\right)\ne y\left\}\right)=\ell \left(h\right)$ (in fact, this or something similar is probably how it should be formally defined), and therefore the above sum equals ${\sum }_{k=1}^m\frac{1}{m}\ell \left(h\right)=\ell \left(h\right)$.