If a balloon is half deflated it goes limp and holds no shape at all. We try playing with it, but no amount of squeezing gets it to take the shape we would need. A balloon that is too inflated has the opposite problem. The shape is there, but it is stretched so tight that it becomes “unstable”, to the point where the slightest squeeze risks bursting it.

These are the two extremes we started from in the first part, underfitting and overfitting, looked at from the side of how well a system manages to settle. The “deflated” model has not reached any “shape” close to the one described by the data. The model that has “filled out” the shape of the data perfectly, noise included, now has no give left for taking in a new datum.

Knowing how much to inflate is not something you decide by eye. You need an instrument, and in the first part the instrument was a pressure gauge. It allowed us to read the error and to break it down into three pieces that add up, namely variance, bias and noise. We saw a few tricks of mathematical magic and we observed how in the end there is a way to measure the error. Even so, it remains impossible to reach what we might call an “ideal state”.

Now, though, we have a problem. The ideal function $\mathcal{I}(x)$ represents the average of the output over the noise, while the typical prediction $f_{\mu}(x)$ is the average of the prediction over the datasets. Every center was a center of mass, and the trick of mathematical magic worked precisely because the reasoning turned around the average.

But in a classification model all this is no longer enough. Having to predict categories, we cannot freely add and divide the values of the output. It goes without saying that there is not much sense in computing “the average between shoes and boots” or “a shoe divided by three”.

From the point of view of a gas, so far we have been reading the average position of a molecule, which is a continuous quantity. Now we read a discrete label of the kind “is the balloon’s pressure within the critical threshold?”. There is no midpoint in this measurement, only the labels “within the threshold” and “past the threshold”.

We have to change something, and this change is the passage from the mean to the mode. If in the first case what interests us is the center of mass of the values, now what interests us is the value that occurs most often.

Chimeras are out of mode

One solution might be to assign the pair $(0,1)$ to the comparison between two categories. But the result would not be a category. For example, if a model half the time picks lion and half the time goat, the average will be $0.5$. In practice we get a chimera.

We talked about the center of mass and about how its moving outside the set of possible answers makes the scaffolding that held up the decomposition collapse.

Using the pressure gauge we were reading the average position of a molecule which, as we know, is a continuous quantity. Now we need a warning light that stays off if, while inflating the balloon, we are within the critical threshold, and comes on if we go past it.

On the left the graduated dial, where the needle indicates a value along a continuous scale. On the right the same instrument with no graduation, carrying only the threshold mark in red and the warning light above it. The latter is off because the pressure is still below the safety threshold. Same pressure, two different readings, one continuous and one with only two labels.

Given a certain number of measurements, the mode identifies the value that occurs most often, and it is exactly the mathematical tool we need. Its usefulness lies precisely in keeping intact the set over which the categories to be classified are defined. Which also represents its greatest limitation, precisely because it is an operation that cannot be reconciled with additivity.

Pedro Domingos, in his 2000 paper (“A Unified Bias-Variance Decomposition”), described exactly this attempt to extend the bias-variance decomposition to the case of classification. The starting point is the choice of the 0-1 loss, which returns $0$ if the predicted class is right and $1$ otherwise.

Despite the name, the 0-1 loss should not be confused with the cost functions used in training. Among the various aspects, what certainly stands out is that it is not differentiable, and that it is therefore essentially inapplicable to the backpropagation stage. On the contrary, in validation and testing it provides in a simple and direct way the percentage of errors the model has made. Exactly what a warning light is supposed to do.

Loss functions according to their role:

training stage: they are differentiable functions (e.g. Cross-Entropy) that allow the gradients to be computed during backpropagation.

validation stage: at every epoch they monitor the actual accuracy in order to tune the hyperparameters.

test stage: they measure the effective error of the trained model, after the training stage.

Let us start again from the fact that, by varying the dataset $\mathcal{D}$, we obtain a new model. So let us take up our three actors again and redefine them with respect to the mode.

  • $\mathcal{I}(x)$: it remains the ideal to be reached. The difference is that now it does not return a value but a category.

  • $f_m(x)$: it represents the dominant prediction, the class the model produces most often as the dataset varies. It takes the place of $f_{\mu}(x)$ as the center of the predictions.

    $$ f_m(x)=\arg\max_y P_{\mathcal D}\left(f(x,\phi_{\mathcal D})=y\right) $$

  • The bias is measured through the 0-1 loss, which signals whether or not the prediction coincides with the ideal value. Using the notation of indicator functions:

    $$ \text{bias} = \mathbb{1}\big(f_m(x) \ne \mathcal{I}(x)\big) $$

    Or written another way:

    $$ \text{bias} = \begin{cases} 0 \quad \ : f_m(x) = \mathcal{I}(x) \\ \\ 1 \quad \ : f_m(x) \neq \mathcal{I}(x) \end{cases} $$

The variance now describes how likely it is that a trained model produces a prediction different from the dominant prediction, obtained by considering all the possible datasets:

$$ \text{variance} = P_\mathcal{D}\big(f(x,\phi_\mathcal{D}) \ne f_m(x)\big) $$

From this definition follows an important property that we will use later on.

By construction, $f_m(x)$ is the most frequently predicted class by the models trained on all the possible datasets. In the case of binary classification, consequently, a model trained on a random dataset coincides with $f_m(x)$ at least 50% of the time.

This implies that the probability of obtaining a prediction different from $f_m(x)$, that is the variance, cannot exceed $0.5$:

$$ \text{variance} = P_{\mathcal{D}}\big(f(x,\phi_{\mathcal{D}})\neq f_m(x)\big) \leq 0.5. $$

The error we want to decompose is instead the probability that the model assigns to $x$ a class different from the ideal one:

$$ \text{Err}(x) = P_{\mathcal{D}}\big(f(x,\phi_{\mathcal{D}})\neq \mathcal{I}(x)\big). $$

Starting from this quantity we will show how the error can be expressed as a function of the bias and of the variance.

We have already talked about how, at the microscopic level, every molecule of a gas has a different speed and continually changes direction because of collisions. It would be impossible to describe the motion of every single particle.

Statistical mechanics gets around the problem by introducing two concepts:

  • an average quantity, for example the average speed of the molecules;
  • the fluctuations around that average, which measure how far the speeds of the individual particles depart from the average value.

In regression, the variance measures how much the predictions of the different models oscillate around the average prediction. In classification, since there is no average between categories, the role of the “average speed” is taken on by the dominant prediction $f_m(x)$, while the variance measures with what probability a model trained on a different dataset produces a class different from the dominant one.

So the variance does not measure how much a model gets wrong, but how much its predictions fluctuate when the training dataset changes, just as in physics the variance of the speeds does not measure whether a molecule is “right” or “wrong”, but how far it departs from the average behavior.

Tertium non datur

Let us take the simplest case, with only two classes and no noise. Here everything turns around an elementary observation. If a prediction does not coincide with one class, then it coincides with the other. This is a binary classification and therefore there are no third possibilities.

The model’s error is determined by the interplay of two aspects. The correctness of the dominant prediction, and the possible departure of the individual model from that prediction. Depending on whether or not $f_m(x)$ coincides with the ideal class, we get two distinct situations.

For brevity we write $f$, $f_m$ and $\mathcal{I}$ without their argument.

Case A: dominant prediction right ($f_m = \mathcal{I}$)

Just as in a gas at equilibrium, the most probable configuration coincides with the one observed at the macroscopic level. The deviations of the individual models represent mere fluctuations around this state. The only source of error remains the fluctuation of the individual realizations around the system’s most probable state, that is the variance.

Essentially this means that disagreeing with the ideal value is exactly the same as disagreeing with the dominant prediction:

$$ f \ne \mathcal{I} \iff f \ne f_m $$

Consequently, the only way the model can go wrong is through a fluctuation with respect to the dominant prediction:

$$ \text{Err} = P_\mathcal{D}(f \ne f_m) = \text{variance} $$

Case B: dominant prediction wrong ($f_m \neq \mathcal{I}$)

The system’s most probable configuration does not coincide with the correct one. By analogy with a gas, it is as if the most frequently observed macroscopic state did not represent the value we want to describe. The individual realizations can, however, still depart from this dominant configuration.

Since we are considering a two-class problem, if the majority lies on the wrong class, the only possible alternative is the correct class $\mathcal{I}$. So a fluctuation with respect to the dominant configuration necessarily leads to the right prediction:

$$ f = \mathcal{I} \iff f \neq f_m $$

In statistical terms, the probability that an individual realization departs from the dominant configuration coincides with the probability of obtaining the correct class:

$$ P_\mathcal{D}(f = \mathcal{I}) = P_\mathcal{D}(f \ne f_m) = \text{variance} $$

The classification error is therefore the complement of this probability:

$$ \text{Err} = P_\mathcal{D}(f \ne \mathcal{I}) = 1 - P_\mathcal{D}(f \ne f_m) = 1 - \text{variance} $$

In this case we obtain:

$$ \text{bias} = 1 $$

It follows therefore that:

$$ \text{Err} = \text{bias} - \text{variance} $$

If the system’s most probable state is far from the correct state, the fluctuations of the individual particles can carry some results in the right direction. Here the model’s variability, instead of increasing the error, makes it possible to compensate for a systematic error of the dominant configuration.

How the models' predictions are distributed over the two classes. The green part is the correct class, marked by $I$, the red part is the wrong one. Below each bar, $f_m$ points to the larger segment, that is the class the model predicts most often. In Case A, at the top, the two symbols line up on the same side. In Case B, at the bottom, they fall on opposite sides, and the whole difference between the two cases lies there.

A + B

The two cases just analyzed can be brought together into a single expression. To do so we need a coefficient that takes on different values depending on the bias.

The term:

$$ 1 - 2 \text{ bias} $$

has exactly this property. When the dominant prediction coincides with the ideal class and $\text{bias} = 0$, it equals $+1$. When instead the dominant prediction is wrong and $\text{bias} = 1$, it equals $-1$.

We can therefore write the decomposition of the error as follows:

$$ \text{Err}(x) = \text{bias} + (1 - 2 \text{ bias}) \cdot \text{variance} $$
case bias error effect of the variance
A, dominant prediction right $0$ variance makes it worse
B, dominant prediction wrong $1$ $1 -$ variance makes it better

The interesting aspect is that the sign of the variance depends on the bias. If the dominant prediction is correct, the variance simply represents a fluctuation that pulls some models away from the right answer, increasing the error. If instead the dominant prediction is wrong, those same fluctuations allow some models to reach the correct class, reducing the error.

Taking up the analogy with statistical mechanics again, the variance always describes the fluctuations around the system’s most probable state. When that state coincides with the correct configuration, the fluctuations are unwanted. When instead the dominant state is the wrong one, the fluctuations become the only mechanism that allows a trained model to get closer to the correct configuration.

This is the main difference with respect to the case of regression seen in Part 1. There, bias and variance always contributed with a positive sign and could be analyzed separately. In classification, on the other hand, bias and variance are coupled. The contribution of the variance depends on the value of the bias and can no longer be studied independently of it.

The height of each bar is the error, the number written at the bottom is the variance. In the group on the left the bias is zero and the bars rise as the variance grows, so the error increases. In the group on the right the bias equals one and the same variances give lower and lower bars, so the error decreases. In the last column both groups reach 0.5, a value neither of them can go past.

It is worth recalling that for simplicity we started from the assumption of no noise, so everything we have written so far describes a system in which the only uncertainty is the one introduced by the dataset. When noise is there, and in practice it always is, the correct label becomes uncertain too and its uncertainty becomes part of the decomposition. In that case as well it does not add up as a term of its own, but interacts with the other two in the same way we have just seen bias and variance interact.

Sometimes instability pays off

Case B tells us something that, with the squared loss of regression, would seem absurd. If the model is systematically wrong, its instability can actually reduce the error.

A balloon shows this better than algebra does.

The same balloon with the same average pressure, just below the threshold. On the left the gas is still, the needle does not move and the warning light stays off the whole time. On the right the gas is shaken, the yellow needle oscillates and some readings go past the threshold mark, so the light comes on every now and then. The average has not changed, what has changed are the fluctuations.

Let us go back to our balloon full of gas and to the warning light that has to detect when the pressure goes past a certain threshold. Because of a faulty calibration, the balloon’s average pressure sits just below that threshold. If the gas were perfectly still, the measurement would always be the same and the light would stay off the whole time, so the system would be stable but always in error.

Now let us shake the balloon. The gas molecules strike with greater intensity and the pressure starts to oscillate around its average value. Some oscillations go past the threshold and the light comes on for a few instants. It is the fluctuations of the gas that occasionally carry the measurement over to the right side, even though the average pressure has not changed.

If the dominant prediction $f_m$ is wrong, a perfectly stable model would go on repeating the same error forever. The variance, on the other hand, introduces fluctuations into the predictions. Some models move away from the dominant prediction and, precisely for this reason, end up on the correct class.

In other words, an unstable model affected by a strong bias can make fewer errors than a stable model with the same bias. The variance does not eliminate the bias, but in this case it softens its effects, just as the fluctuations of the gas allow the pressure to cross, every so often, a threshold that the average value would never reach.

Let us work through an example so as to make the phenomenon clearer. Let us consider an instance $x$ for which the dominant prediction is wrong ($\text{bias}=1$) and suppose that the correct class is $C$, while the dominant class is $S$, that is the wrong one.

To understand why an increase in the variance can reduce the error, let us imagine at first that the predictions are strongly concentrated on class $S$.

We might have, for example:

$$ P(S)=0.95,\qquad P(C)=0.05 $$

In this case the model predicts the wrong class 95% of the time and gets it right only 5% of the time. If, however, the dispersion of the predictions increases, they become less concentrated around the dominant configuration and a larger share of the probability can shift toward the correct class.

For example:

$$ P(S)=0.70,\qquad P(C)=0.30 $$

Continuing to increase the dispersion, we can even approach the limiting case in which the two classes become equally probable:

$$ P(S)=0.50,\qquad P(C)=0.50 $$

In this situation the model is no longer systematically oriented toward the wrong class, coming to behave, in the limiting case, like the toss of a coin, and the error drops to 50%.

The three cases just seen. The part in red is the share of predictions falling on the dominant class $S$, that is the wrong one. The green part is instead the one falling on the correct class $C$. As the dispersion grows the red part gets shorter and the green one gets longer. The dashed line marks the value 0.5, below which the red part cannot go, because $S$ remains in any case the most frequent class.

The opposite reasoning holds as well. In Case A, where the dominant prediction coincides with the correct class, we have

$$ \text{Err}=\text{variance}\le 0.5. $$

Here the fluctuations can only make performance worse, but not even in the most extreme case do they manage to push the error beyond 50%. Even an extremely unstable model in fact goes on agreeing with the dominant prediction at least half the time.

The two cases meet at exactly this point, 50%. It is the limit imposed by binary classification. On one side it represents the best result obtainable when the majority is wrong, on the other the worst result when the majority is right. As already said, we arrive at a state equivalent to a coin toss, where the model no longer contains any useful information about the correct class.

The moral, then, is this. The decomposition of the error into variance, bias and noise always exists, but it does not always take the same form. And above all, in cases like this one, it certainly does not take an additive form.

Additivity is a privilege of the squared loss and depends neither on the model nor on the data, but only on the geometry of the square, that is on the fact that the mixed term vanishes after the expansion.

When we change the measure of the error, the way these three contributions combine with one another changes as well.

Probability is common sense reduced to calculation

I think that good old Laplace (one of the most important French scientists of the Enlightenment) gathered into a single sentence one of the aspects I personally find most curious about the current discussion around AI. On one side we keep trying to “capture” inside a language model some likeness to the human brain, on the other we try to make inference more and more correct. But if in the end what we are after is for a model to reach “common sense”, let me point out that the world is already full of intelligences capable of telling whether an event is more or less plausible.

But let us go back to our problem. What is certainly a matter of “sure common sense” now is to change the way we evaluate the output. It is no longer worth asking “lion or goat?”, but “how sure are you that it is a lion?”.

The Brier score does exactly this.

The model produces a probability:

$$ \hat{p} \in [0,1] $$

The true label for the comparison will be:

$$ y \in \{0,1\} $$

The score will finally be computed like this:

$$ \mathcal{L}(x) = (\hat{p} - y)^2 $$

Does it not remind you of something we have already seen? The Brier score is essentially a squared loss, only computed on probabilities. And being squared it decomposes exactly as we have already seen, into variance, bias and noise.

Additivity comes back, and with it the trick seen in the first part.

This is why, when one really wants to understand the behavior of a classifier and reason rigorously about bias and variance, accuracy alone is not enough. A measure that returns only a right or wrong result in fact risks losing a great deal of important information. It is often more useful to employ metrics that also take into account how sure the model is of its own predictions, such as the Brier score and cross-entropy (introduced in the article about coffee).

The practical advantage of that sum lies precisely in the fact that each term can be modified without altering the others, so we can act on one component at a time. They are the three quantities that govern the behavior of our system, each one associated with a different limit of learning.

The temperature cannot be lowered. It is the noise $\sigma^2$, the intrinsic thermal agitation of the datum. The same input can correspond to different outputs, and no model, however perfect, can predict them all. It is the unbreachable wall of learning, the one we already ran into in Part 1.

The molecules observed at any one instant are the data. The variance arises from the fact that the model is trained on a finite and inevitably noisy sample, and therefore changes every time the sample changes. The reason why having more data helps is exactly the reason of the pressure gauge. With few molecules the needle wavers, with many molecules it settles because the impacts compensate for one another. Every anomalous point weighs less and less and models trained on different datasets end up resembling one another more and more. The term $\mathbb{E}_{\mathcal{D}}[(f-f_\mu)^2]$ measures precisely this difference between models and tends to contract.

The volume represents the capacity of the model. The bias arises from the rigidity of the chosen family of models. The “ideal shape” the model wants to reach might simply not be among those the model itself is able to take on. A gas confined in a volume that is not very large can explore few configurations, whereas if we consider a larger volume then the number of possible configurations grows and the average prediction $f_\mu(x)$ can get closer and closer to the ideal $\mathcal{I}(x)$.

Un colpo al cerchio e uno alla botte

“Un colpo al cerchio e uno alla botte”, literally “one blow to the hoop and one to the barrel”, is said in Italian of anyone who has to keep two opposing demands in balance, giving something to each in turn.

The saying comes from the old trade of the master coopers who, while making wooden barrels, had to know how to land their mallet blows properly. One blow to the metal hoop to make it slide and set it in place, one blow to the staves of the barrel to settle them and keep the wood from splitting.

Dealing with bias and variance is not very different. We can increase the capacity of the model so as to lower the bias, but in exchange we would get a higher variance. This is because a model has no way of telling signal from noise. Therefore the very freedom that allows the model to follow the correct “ideal shape” ends up allowing it to fit just as effectively the random fluctuations present in the samples it has “observed”.

We can describe this relationship by looking at the reducible error as a function of the capacity of the model:

$$ \text{reducible error} = \underbrace{ \mathbb{E}_\mathcal{D}\big[(f-f_\mu)^2\big] }_{\text{grows with capacity}} + \underbrace{ \big(f_\mu(x) - \mathcal{I}(x)\big)^2 }_{\text{shrinks with capacity}} $$

At the beginning, when the model has little capacity, it is too simple to represent the signal well. It follows that the main problem is the bias. Increasing its capacity makes it possible to get progressively closer to the correct function and the error decreases.

Past a certain threshold, however, the variance comes into play. The model is by now flexible enough to represent the signal, but precisely for this reason it can start to follow the accidental fluctuations of the sample as well. The error therefore goes back to growing.

The result is the classic U-shaped curve:

  • low capacity $\Rightarrow$ high bias, low variance $\Rightarrow$ increasing capacity improves the model
  • intermediate capacity $\Rightarrow$ bias and variance reach a compromise $\Rightarrow$ the optimal point is found
  • high capacity $\Rightarrow$ low bias, high variance $\Rightarrow$ increasing capacity further worsens generalization.

From left to right the capacity of the model grows. The two dashed curves are the components of the reducible error, the bias squared which decreases and the variance which increases. The solid curve is their sum and the marked point is its minimum, that is the capacity that makes the error as low as possible.

If you do not expect the unexpected you will not discover it

The Greek philosopher Heraclitus finished this sentence with “…being as it is untrackable, and inaccessible”, which could make it more fitting than ever for these times of AI as well.

What we can certainly say is that something similar happened in the case of the phenomenon we are about to see.

According to classical theory, then, there exists an optimal capacity beyond which an ever more powerful model should generalize ever more poorly.

At a certain point, however, probably during an almost serendipitous event, people began to observe that this U-shaped curve does not always go on behaving as expected. In some cases, after the model has reached a peak, which lies precisely around the interpolation threshold (the point at which the model becomes powerful enough to “memorize” all the training data perfectly), the error goes back down, reaching even below the minimum observed in the classical curve.

This phenomenon was formalized for the first time in the 2019 paper Reconciling Modern Machine Learning Practice and the Classical Bias-Variance Trade-off (Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal). A few months later, a group of researchers at OpenAI extended the discovery by showing that the phenomenon does not occur only as a function of the size of the model, but can also show up during training, as the number of epochs varies. This latter study, titled Deep Double Descent: Where Bigger Models and More Data Hurt (Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever), helped to spread the term double descent.

We can describe double descent intuitively by observing how the error varies as the capacity of the model grows:

  • Small model: underfitting $\Rightarrow$ high error

  • Medium model: good generalization $\Rightarrow$ low error

  • Large model: we are close to the interpolation threshold $\Rightarrow$ high error

  • Even larger model: double descent $\Rightarrow$ the error goes back down

In other words, as the capacity of the model increases, the error can first decrease, then increase in the interpolation region and finally decrease again when the model becomes larger still.

It is worth looking carefully, though, at which of the four steps is really the anomalous one. The first three are not anomalous at all, because they describe exactly the U-shaped curve we have just derived from the compromise between bias and variance. In the fourth, past the interpolation threshold, the variance ought to go on growing together with the capacity. And instead no, it stops doing so. Surprise!

The curve carries on past the interpolation threshold, represented by the vertical line in red. The grey dashed line shows how the curve ought to behave according to classical theory, that is going on with a strictly increasing trend. This one instead, having reached a maximum point, immediately goes back down and reaches a new minimum with respect to the one it had touched before the threshold.

At this point, however, it is important to clarify what memorizing the data really means and why this does not necessarily imply bad behavior on the test set.

Taking up again what was written in the article about neural networks, the fact that the training loss gets close to zero means that the model manages to pass practically through all the observed points. However, there exist infinitely many functions capable of passing through the same points. These functions are indistinguishable on the training data, but they can behave very differently in the spaces between one point and the next. It is precisely this behavior on unobserved data that determines whether the model will generalize well or badly.

The same observed points, crossed by three different functions. All of them pass through every observed point, but they behave very differently in the spaces in between. The green curve is regular, the other two oscillate. On the training data they turn out to be indistinguishable, whereas on new data they are not.

So, saying that a model has memorized the data is not enough to understand which function it has actually learned. The crucial point is to understand which of the infinitely many possible solutions is chosen by the model.

This is where the concept of inductive bias comes in. In this case the reference is not to the bias-variance decomposition, but to a kind of bias that indicates a tendency of the model to prefer certain solutions over others. For example, a model may have a preference for simpler or more regular functions, even when there exist a great many functions equally capable of achieving a training loss close to zero.

This also helps in understanding the double descent seen earlier, since going past the interpolation threshold and reaching an optimal training loss does not necessarily mean that the model will generalize worse. Even among the models that memorize the data perfectly, some can find solutions that generalize very well.

In a gas, counting the functions compatible with the data is the same as counting the microstates compatible with the constraints (this subject too was covered in the article about coffee). The over-parameterized model is a gas with too much volume at its disposal. What makes it settle on an optimal configuration rather than an erratic one?

We will try to answer this question in the third part.

Key terms

  • Mode: the value that occurs most often in a set of observations. In classification it replaces the mean as the reference centre of predictions.
  • 0-1 loss: a function that returns $0$ if the predicted class is correct and $1$ otherwise. It is not differentiable and is not used in training, but it is straightforward to interpret in validation and testing.
  • Dominant prediction $f_m(x)$: the class the model produces most often as the training dataset varies. It corresponds to the mode of the predictions.
  • Bias (classification): equals $0$ if the dominant prediction coincides with the ideal class, $1$ otherwise.
  • Variance (classification): the probability that a model trained on a different dataset produces a prediction different from the dominant prediction. It cannot exceed $0.5$.
  • Brier score: a squared loss computed on the predicted probabilities rather than on continuous values. It restores the additivity of the bias-variance decomposition to the classification setting.
  • U-curve: the behaviour of the reducible error as a function of model capacity. It falls with bias when capacity is low, and rises with variance when capacity is too high.
  • Interpolation threshold: the point at which the model’s capacity becomes sufficient to pass through all training points with zero loss.
  • Double descent: the phenomenon whereby the error, after reaching a peak near the interpolation threshold, falls again in overparameterised models.
  • Inductive bias: the tendency of a model to prefer certain solutions over others among the infinite functions compatible with the training data.