Given a trained model, when it gets something wrong, how do we describe the error it has made? In Part 1 we answered this question by breaking the error down into the sum of three clearly distinct parts. The bias depends on the model itself. The variance, on the other hand, depends on the fact that the model was trained on a finite sample and therefore changes as the composition of the dataset changes. And finally there is the noise, an element that is part of the data being used and cannot be removed in any way. The result we obtained from this decomposition lies precisely in its additive nature. A nature that emerges from the computation of the mean and that allows us to study these three pieces one at a time.
In Part 2 we put what we had learned to the test in a new setting. Moving from regression to classification, we saw how additivity no longer applies. There is no mean between categories, and for this reason one has to reason in terms of the mode. What came out of it is that the error can still be written as a function of bias and variance, but not as a sum, and that the contribution of the variance changes sign depending on the bias. If the dominant prediction is wrong, an unstable model gets things wrong less often than a stable one.
What do the first two parts of this article have in common? Measurement. In both cases an already trained model is taken into examination, the error is observed and it is broken down. At the end of the second part, with double descent, we also anticipated that observing the error depends on the way we assess the model’s ability to generalize.
What is missing? An instrument such as a pressure gauge reads the pressure, it does not change it. Neither of the two parts tells us how to act on the conditions that produce the error we have read.
We can introduce the last subject of this series with a single word: regularization.
From balloons to pistons
In the first two parts the gas sat in a balloon, and that was the right object as long as it was a matter of measuring. A balloon, however, has one flaw for what we have to do now, namely that its volume depends on how much gas we put inside it. It cannot be enormous and nearly empty, because without enough gas it goes limp. Here we need something that represents the condition in which very few molecules wander about in a boundless space.
Thermodynamics textbooks often use a cylinder with a piston as an example. And it is an image that can come in useful here, because the volume is decided by the piston and is not determined directly by the gas it contains.
In the two balloons on the left the amount of gas is different and the volume adjusts accordingly, because a balloon with little gas goes limp and holds no shape. In the two cylinders on the right the molecules are instead the same in both cases, and what establishes the volume is only the position of the piston.
Part 2 closed on a question. An over-parameterized model can take on infinitely many configurations, all of them compatible with the data, that is to say it is a gas with too much volume at its disposal, and yet something makes it settle on a regular configuration rather than an erratic one.
Before saying what that something is, we have to understand how large that volume really is. The answer is interesting, and it is the real reason why regularization is indispensable.
Let us also take up again the image of the model passing close to the observed points. The behavior it takes on is not a marginal detail at all. In high dimension the observed data in fact occupy a minuscule portion of the space of possible configurations. Most of the space lies “between” the data. It is precisely in this region, where we have no direct observations, that the “shape” taken on by the model emerges.
Let us work through a numerical example. Let us take a problem described by $40$ variables, that is a space with $40$ dimensions, and let us divide each dimension into $10$ intervals. Nothing particularly precise, it only serves to give an order of magnitude. From this partition we obtain $10^{40}$ possible regions of the space.
Note the point carefully: the dimensions are established by the variables that describe the problem, whereas the number of available examples only tells us how far we manage to populate that space. And it is from the comparison between the two that the difficulty arises.
If we have $10^4$ examples, that is ten thousand points, on average we have a single datum every
$$ \frac{10^{40}}{10^{4}} = 10^{36} \quad \text{regions.} $$Most of the space therefore remains completely devoid of observations. The few available points are scattered and isolated within an enormous set of regions that have never been observed. It is precisely in these regions that the model extends the structure learned from the available data, and this extension plays a crucial role in determining its performance on new data. In Machine Learning the phenomenon goes by the name of curse of dimensionality.
Eight observed data points placed in a space with one, two and three dimensions. With every dimension added the number of regions is multiplied by ten, while the available data remain the same. With forty dimensions the regions become $10^{40}$.
Let us try to look at the problem from the point of view of the piston. Let us imagine it pulled back almost to the end of its travel. A rarefied gas, under these conditions, occupies a tiny portion of the enormous volume available. Its molecules can arrange themselves in a boundless number of configurations, all of them equally compatible with the macroscopic state we observe. What we are observing, in other words, is a condition of maximum uncertainty. And in this state of affairs, pushing the piston becomes an action aimed at reducing the space and making the system more predictable.
Regularization in fact has the same task. And it carries it out by taking into consideration the two phenomena that determine the gap between training and test: overfitting and the presence of unconstrained regions.
In the first case, the model does not limit itself to representing the structure of the phenomenon, but fits the training data to the point of incorporating their accidental peculiarities and the noise attached to the individual samples as well. The learned representation thus becomes too specific to the observed examples and loses its ability to generalize.
The unconstrained regions, on the other hand, are the portions of the space in which the training data are not present, or are extremely scarce. Here the observations provide little or no direct information about the behavior the model ought to take on. Consequently the model can produce a great variety of solutions, all of them compatible with the observed data but very different from one another. In high dimension this phenomenon becomes particularly relevant, because the data tend to be extremely scattered, and so even a very large training set can leave most of the space devoid of observations.
On the left the red curve passes through every observed point and takes in the noise as well, moving away in this manner from the dashed function it should have approximated. On the right the grey bands cover the intervals with no observations, in which the data impose no behavior at all on the model.
The solution to look for is therefore to have regularization bring about a narrowing of the space of possibilities, “pushing” the model toward the solutions that display better generalization properties. Toward solutions, in other words, that are more stable, that vary more gradually and that depend less on the details of the individual samples.
Put another way, regularization keeps the model from spreading out freely into the regions unconstrained by the data and limits its degrees of freedom. Just as with our piston, we do not impose where every particle has to be, we limit the space within which it can spread out.
Everything should be made as simple as possible, but not simpler
This sentence, commonly attributed to Einstein, is a good excuse for adding a consideration to the matter of regularization. Regularization in fact arises from the need to limit the complexity of the model without compromising its ability to generalize.
A note on the quotation. The formulation we all know does not appear in any of Einstein’s writings. It actually derives from a paraphrase that Roger Sessions published in 1950 of a remark made by Einstein in 1933, decidedly more long-winded and more cautious. Which, for a sentence about the risks of excessive simplification, I would say fits perfectly.
We will see how to arrive at this result by starting from a change made not to the network, but to the criterion by which we judge it during training: the loss.
Adding terms to the loss
And here is the loss:
$$ \hat\phi = \operatorname*{argmin}_{\phi} \big[\,L(\phi)\,\big] = \operatorname*{argmin}_{\phi} \left[\sum_{i=1}^{N} \mathcal{L}\big(f(x_i,\phi),\,y_i\big)\right] $$The notation $\operatorname*{argmin}_\phi$ indicates the value of $\phi$ that makes the expression minimal, while the summation is the total loss, that is the sum of the errors made by the model $f(x_i,\phi)$ over all $N$ examples.
In order to “push” the model toward certain solutions we add the term $g(\phi)$ and the coefficient $\lambda$:
$$ \hat\phi = \operatorname*{argmin}_{\phi} \left[\sum_{i=1}^{N} \mathcal{L}\big(f(x_i,\phi),\,y_i\big) + \lambda\cdot g(\phi)\right] $$The term $g(\phi)$ takes on high values for the configurations of the parameters we want to penalize and lower values for the ones we want to favor. The coefficient $\lambda$, which we assume to be positive, determines instead the weight of the preference introduced by $g(\phi)$ relative to adherence to the data.
If it is small, adherence to the data prevails; if it is large, the importance of the preference expressed through $g(\phi)$ grows.
We can therefore interpret $g(\phi)$ as a restoring force, while $\lambda$ determines its intensity. It is the piston moving forward and reducing the volume available to the molecules. In the same way the regularization term pushes the parameters toward the favored region, narrowing the space the gas can occupy. The coefficient $\lambda$ is the force with which we press on the piston.
The three circles represent the space the parameters can occupy for increasing values of $\lambda$. The arrows are the restoring force exerted by $g(\phi)$ and their length grows together with $\lambda$. The more this force increases, the more the parameters stay confined close to zero.
Now, which configurations to penalize and which to favor is a choice we make ourselves. Through $g(\phi)$ we are in other words introducing into the model a preference about the solution to be learned. One question remains, though. Is there a reason to prefer certain parameters, or are we simply acting arbitrarily?
The answer comes from looking at where losses come from, that is from the criterion of maximum likelihood.
$$ \hat\phi = \operatorname*{argmax}_{\phi} \left[\prod_{i=1}^{N} P(y_i \mid x_i, \phi)\right]. $$Here $P(y_i\mid x_i,\phi)$ represents the probability the model assigns to the observed output $y_i$, given the input $x_i$ and the parameters $\phi$. In other words, it measures how plausible the model considers the correct answer to be for that particular example.
Multiplying these probabilities over all the examples of the dataset we obtain
$$ \prod_{i=1}^{N} P(y_i\mid x_i,\phi). $$This product measures how compatible the entire dataset is with the model. If the model assigns high probability to the observed answers, the likelihood is high. If instead it assigns low probability to many of them, the likelihood turns out to be low. This is why the product goes by the name of likelihood of the dataset.
Let us now add a prior distribution $P(\phi)$. This distribution describes how plausible we consider the different configurations of the parameters $\phi$ to be before observing the data. In this way we can incorporate into the model a preference about which solutions we deem more reasonable.
The criterion becomes precisely
$$ \hat\phi = \operatorname*{argmax}_{\phi} \left[P(\phi)\prod_{i=1}^{N} P(y_i \mid x_i, \phi)\right]. $$The first factor, $P(\phi)$, expresses what we were assuming about the parameters before seeing the data. The second, as we have already seen, measures how well those parameters explain the observed data. The product therefore brings together what we deemed plausible in the first place with what the data show us.
This criterion is called maximum a posteriori, or MAP. The solution chosen is the one that, combining prior information and observed data, turns out to be the most plausible after having seen the dataset.
Taking finally the negative logarithm, the product turns into a sum and the maximization problem becomes a minimization problem. In this way the probability term can be expressed as a regularization term, according to the relation
$$ \lambda\cdot g(\phi) = -\log P(\phi). $$The regularization term is therefore the negative logarithm of a prior distribution over the parameters. Which means that penalizing certain parameters amounts to declaring that a priori we deem them improbable.
It is the same logarithmic structure that connects entropy and probability in the piece about coffee, where the negative logarithm of a probability turns up again as a cost.
So, in the end, choosing a regularization term means establishing which values of the parameters we consider plausible before observing the data. And probability theory provides exactly the language and the tools needed to formalize this kind of prior assumption.
L2 regularization (Weight Decay)
So far $g(\phi)$ has remained an abstract term. The most widespread form is also the simplest. If we want to penalize the parameters that take on large values, it is enough to act on the sum of their squares.
$$ \hat\phi = \operatorname*{argmin}_{\phi} \left[\sum_{i=1}^{N} \mathcal{L}\big(f(x_i,\phi),\,y_i\big) + \lambda \sum_{j} \phi_j^2\right] $$Here the index $j$ runs, as one would expect, over all the parameters. This is the L2 norm, which in the literature also appears as Tikhonov regularization or ridge regression. When the parameters of a layer are arranged in a matrix, this penalty coincides with the square of the Frobenius norm of the weight matrix, which is therefore the same object written in matrix notation.
The link with the previous section is immediate. The negative logarithm of a Gaussian centered at zero in fact contains precisely a quadratic term. For this reason, introducing an L2 penalty amounts, from a probabilistic point of view, to assuming that the parameters of the model $\phi_j$ follow a priori a Gaussian distribution with zero mean. Small values are therefore considered more plausible, while very large ones become progressively less probable. In this reading $\lambda$ also acquires a precise meaning, because it is inversely proportional to the variance of that Gaussian: the narrower the prior bell, the stronger the penalty.
This can be seen directly in the formula, where the term $\lambda\sum_j \phi_j^2$ grows the further the parameters move away from zero. For example, if a parameter equals $\phi_j=1$, its contribution to the penalty is $1^2=1$. If it equals $\phi_j=5$, the contribution becomes $5^2=25$. For the same error on the data, the second model is therefore penalized far more. L2 is thus not a choice made out of habit, in that it expresses the fact that a good model should manage to generalize without having to resort to overly large parameters.
From small weights to a regular function. In a network the term applies to the weights and not to the biases (here meaning the intercept of the linear function), and this is where the name weight decay comes from. Every layer computes its own output as a weighted sum of the activations it receives, so with small weights the output varies little even when the input varies a great deal. The same holds for every layer, and the effect accumulates all along the network. In the extreme case, with all the weights set to zero, the output no longer depends on the input and only the final bias is left, which is the flattest function imaginable.
The effect of $\lambda$. With $\lambda$ small, the regularization term has little bearing and the network goes on fitting all the data points, noise included. As $\lambda$ increases, the network has to find a balance between two opposing demands: staying close to the data and, at the same time, keeping the function regular. Consequently, the sharpest oscillations are progressively damped. If $\lambda$ becomes too large, however, the regularization term prevails over the one for adherence to the data and the function turns out to be excessively flat, to the point of no longer representing the data adequately.
The same observed points, approximated with three increasing values of $\lambda$. In the first case the curve passes through every point and follows all of its oscillations. In the second it reproduces the general trend without chasing the individual values. In the third the penalty prevails and the function becomes too flat to describe the data.
In the language of Part 2 this is a matter of moving along the compromise between bias and variance, one blow to the hoop and one to the barrel. A network that was fitting the data too closely sees its variance go down, because it is no longer forced to pass through every point. In exchange the bias goes up, given that the set of available functions has narrowed to the regular ones.
L2 is therefore compression, that is the act of pushing the piston. The accessible volume shrinks, the molecules, which here are the weights, are forced toward the center and the extreme states become impossible. Large weights correspond to the distant, fast molecules, the ones with violent trajectories. A compressed gas is more ordered and less scattered, and to this corresponds a more regular function.
N.B. This is a very “technical” note, feel free to skip it if you are not interested in the subject.
The effectiveness of $L_2$ regularization as a tool for improving generalization in neural networks has been widely discussed. Although the $L_2$ penalty (or weight decay) favors weights of small norm, the $L_2$ norm of the parameters alone does not necessarily constitute an adequate measure of the functional complexity of the network. In particular, for architectures with positively homogeneous activations, such as ReLUs, the function represented by the network can remain unchanged under suitable rescalings of the weights across the different layers. Consequently, the $L_2$ norm of the individual weights can vary without directly reflecting the complexity of the learned function.
An alternative consists in considering the Lipschitz constant of the network, which quantifies how rapidly the output of the model can vary as the input varies. For a feed-forward network, an upper bound on the Lipschitz constant can be obtained as the product of the spectral norms of the weight matrices. This quantity appears in several theories of generalization bounds. In particular, Bartlett, Foster and Telgarsky (2017) introduced a bound based on spectral complexity, which depends on the product of the spectral norms and on a corrective term tied to the other norms of the weights. The authors also show an empirical correlation between this measure of complexity, the Lipschitz constant and the generalization risk.
How to come down a mountain drunk
Usually, when you come down a mountain on foot while sober, you look carefully at the landscape, you try to work out the direction of the steepest descent, you place your foot in a safe spot, you stop, you recompute and you carry on. It is an effective method but a slow one. And above all it is not perfect, because even the most cautious step has a length of its own all the same. Where the slope is gentle you arrive more or less where you meant to, but where it is steep your foot overshoots the spot you had chosen and you land lower down than expected. The result is that, without having decided to, you tend to avoid the vertical walls and to prefer the broad slopes.
I have been told, on the other hand, that when you throw yourself down the flanks of a mountain (let’s take one in Puglia, a region in the south of Italy where at most we’re talking about hills), perhaps convinced you have seen a grizzly that is not there, your only goal is to reach the valley floor as soon as possible.
In this case, you stagger and take unpredictable steps. And it is precisely this unpredictability that makes the method interesting. If, for example, during the descent you end up inside a small hollow, that is you make an error of route, the next step might get you out of that hollow by pure chance. In this way you can avoid staying stuck in a spot that looks like the valley floor but is not, and go on looking for the real valley, the lowest one. Besides, by moving without constantly stopping to compute the next step, you manage to come down far more quickly.
In short, the “drunken descent” is chaotic, but it is precisely that chaos that makes it effective. The unpredictable steps make it possible to get out of errors by chance and to go on exploring the terrain, while the speed avoids spending too much time looking, step after step, for the perfect direction. And it is something very close to what allows artificial intelligence models to learn quickly without staying stuck in a solution that looks good, but is not necessarily the best one.
On the left the trajectory of the sober descent, made of short, regular steps along the slope. On the right the one of the drunken descent, whose irregular steps allow it to leave the shallow hollow at the top and to reach the deeper minimum.
The idea of the drunken descent is also found in Stochastic Gradient Descent (SGD), an optimization method in which the parameters of the model are updated using, at every step, only a part of the data instead of the whole dataset. This part of the data goes by the name of batch. We speak of “stochastic” precisely because each batch changes at random from one update to the next, introducing a random component into the path of the descent. Consequently, the gradient computed on it varies as well. If we therefore look for a continuous loss that reproduces the average effect of the stochastic updates, alongside the term due to the finite steps a second one has to appear.
Be that as it may, if you have to try something out, do it on paper. Do not try it at home, and above all do not try it on a mountain.
Let us stop for a moment, though, to see where we have got to. In the previous section we wrote a term, we gave it a weight and we added it to the loss. This is called explicit regularization.
As will have become clear, we are now talking about a second method. Some choices we make for practical reasons, and not with the intention of regularizing, end up exerting exactly the same effect. Nobody wrote them into the loss, and yet there they are. This is called implicit regularization.
In the meantime let us see what form is taken by the loss that describes what SGD actually does (while, if you feel like revisiting Gradient Descent in a strange way, you can do so in the article devoted to it: it’s all about origami part 2).
$$ \tilde L_{\mathrm{SGD}}(\phi) = \underbrace{ L(\phi) + \frac{\eta}{4} \left\lVert \frac{\partial L}{\partial\phi} \right\rVert^2 }_{\text{as in GD}} + \underbrace{ \frac{\eta}{4M} \sum_{m=1}^{M} \left\lVert \frac{\partial L_m}{\partial\phi} - \frac{\partial L}{\partial\phi} \right\rVert^2 }_{\text{variation across batches}} $$The symbols. $\eta$ is the learning rate already seen, that is the length of the step. $M$ is the number of batches into which we have divided the dataset, and $m$ indicates one in particular. $L_m$ is the loss computed on that single batch, while $L$ is the one over all the data. The double bars indicate the squared length of a vector, so they measure how far apart two directions are.
The first part of the formula is the cost of the finite steps we were talking about, that is what GD does even without batches. The loss $L(\phi)$ is the descent we thought we were making, and the term with the squared gradient is the preference for broad slopes that we end up following all the same (Barrett and Dherin, 2021). The second part is what SGD adds (Smith, Dherin, Barrett and De, 2021).
To understand what is particular about SGD, let us imagine dividing the dataset into many small groups. GD analyzes them all together and decides which direction to move in. SGD, on the other hand, analyzes one at a time. And every group, naturally, can tell a slightly different story.
One batch might tell us to go to the right, another a bit further upward, another again in a completely different direction. The gradient computed over the whole dataset represents instead a sort of average direction.
The last term of the formula therefore serves to account for this difference. It tells us how far the indications of the individual batches depart from the overall direction. If the batches agree, the term is small. If instead they disagree, it becomes large.
And it is precisely here that an interesting aspect appears. This term does not limit itself to recording the disagreement between the batches, it penalizes it and therefore pushes the model toward the regions in which the groups of data are broadly in agreement about where to go. In the literature this goes under the name of gradient alignment.
The thin arrows indicate the gradients computed on the individual batches, the thick one their average direction. In the case on the left the batches suggest very different directions from one another and the last term of the formula takes on a high value. In the one on the right the directions almost coincide and the term shrinks to nearly zero.
This also helps in understanding why, in practice, SGD can work better than GD and why, often, smaller batches turn out to be more effective than large ones. The smaller the batch, in fact, the more the indications of the individual groups can vary, and consequently this regularization effect increases.
In other words, the noise of SGD behaves a little like the temperature in a gas. A hot system tends not to settle in the deepest and narrowest minimum. The agitation makes it harder to keep there and favors the broader regions of minimum, where the compatible configurations are more numerous.
It is the same principle as the $F = E - TS$ we saw in the piece about coffee. Temperature gives greater weight to the breadth of the regions, that is the entropy, against the depth, that is the energy.
Two minima of equal depth. In the narrow one the oscillations of the system are enough to make it climb back up the walls and to carry it away from the bottom. In the broad one those same oscillations are not sufficient and the system stays there.
At this point, though, it is worth clearing up a possible misunderstanding. Earlier we said that regularizing means compressing the gas, now we are saying that it is worth heating it, and these look like two opposite moves. In reality they are two different ways of acting on the system. L2 acts on the volume, that is on where the weights are allowed to be. SGD acts on the temperature, that is on how agitated the system is while it looks for somewhere to settle. A gas can perfectly well be compressed and hot at the same time, and indeed in practice weight decay and small batches are used together.
This is why the broader regions of minimum tend to generalize better. And this is also why, within certain limits, it is worth “heating” the system by introducing noise or by using small batches.
Stopping at the right moment
A decidedly practical method of regularization consists in interrupting training before it has reached convergence. This method is called early stopping and its effectiveness is due to the fact that the network tends to learn the trend underlying the data first and only afterwards starts to fit the noise. By stopping at the right moment, one manages to capture the ideal trend without taking in the noise.
The green curve is the error measured on the training data and goes on decreasing. The red one is the error on new data, which after a minimum starts growing again because from that moment on the model begins to fit the noise. The highlighted point indicates where it is worth interrupting training.
Early stopping can be interpreted in two ways, which are not necessarily alternatives to one another.
On the one hand, interrupting training while the weights are still relatively close to their initialization introduces a form of implicit regularization very close to L2. In the case of linear regression, for example, stopping gradient descent before convergence keeps the weights from growing excessively and therefore favors solutions with smaller weights, which is exactly what we would obtain by writing a quadratic penalty into the loss.
On the other hand, stopping early limits the effective complexity the model manages to reach. In this reading the number of iterations becomes to all intents and purposes a parameter that controls complexity, in the same way that the number of parameters is one. And this opens a direct link with the phenomenon of double descent with respect to the number of epochs (epoch-wise double descent), in which the generalization error varies non-monotonically as training time grows: it decreases, reaches a peak near the interpolation threshold and, past that threshold, goes back to decreasing. It has to be said that the link between epochs and complexity is not a general explanation of the phenomenon, which can also depend on the way different components of the network are learned at different speeds.
But why, past the interpolation threshold, does the error go back down instead of continuing to rise?
Near the interpolation threshold, the model has a number of parameters barely sufficient to fit the data. Consequently the space of the solutions that describe the dataset well is very narrow. The model has little freedom to choose among different solutions and can be forced to adopt one that is particularly sensitive to the noise present in the data. It is here that this fragility becomes greatest, and from here that the peak of the generalization error arises.
Past the threshold, the scenario changes. The parameters become more than necessary and the solutions compatible with the data become infinitely many. It is then that the problem is no longer finding a solution that interpolates the data, but working out which one, among the infinitely many possible ones, gets selected.
It is worth stressing one point, because it is in fact the answer to the question left open in Part 2. In double descent the selection takes place even when nobody has written a regularization term into the loss. What does the choosing, in that case, is the implicit bias of the optimizer, which among the infinitely many solutions compatible with the data tends to converge toward the one with the smallest norm. The piston, in other words, is not always pushed by us. Sometimes it is already positioned at a given point.
Other practical methods. Among the many, the best known is probably dropout. At every iteration a subset of the hidden units is switched off at random. In the old dense networks about half of them were switched off, whereas in modern architectures much lower rates are used. Every pass therefore uses a slightly different network.
An intuitive way of grasping its effect is to think of a group of programmers working on the same project. If each programmer knows only the files they usually work on, it takes just one of them being unavailable for the work to grind to a halt. If instead everyone knows the project well enough to be able to work even when someone is missing, the system is far more robust.
Dropout imposes something similar on the network. Since at every pass some units are switched off, the network cannot always count on the same units to produce a given answer. It is therefore forced to distribute the information and to build representations that work even when some components go missing.
Then there is data augmentation, which tackles the same problem from the opposite side. Instead of tightening the model, it increases the number of data artificially. From every example others are generated by applying transformations that do not change its meaning, such as rotating or cropping an image. Going back to the count of the $10^{36}$ empty regions, it is the attempt to populate a few more of them rather than forbidding the model to cross them.
Finally ensembling, which gives up on choosing altogether. Several models are trained and their prediction is averaged. In the language of Part 1 it is an approach that bears directly on the variance, because by averaging over several configurations the fluctuations due to the particular training sample tend to cancel out.
All three reduce the risk of the network fitting the details of the training set too closely and favor more robust solutions, capable of generalizing better to new data.
From the Ideal Gas to the Real Gas
We can now say we have everything we need to close this long article. And to do so let us go back over what we have done from the point of view of our cylinder closed by a movable piston. Inside there is a gas, made up of an enormous number of molecules moving continuously in every direction. When these molecules strike the piston, they exert a force that tends to push it outward. The pressure of the gas arises, at the microscopic level, precisely from the sum of these countless impacts.
To describe this system we can start with a very simple model: the ideal gas.
In the ideal gas we make two important simplifications. We imagine that the molecules are so small that their volume can be neglected and we suppose that they exert no forces on one another, except during impacts. In this way the molecules can be considered free to move about inside the container.
The behavior of the gas is then described by the simple relation
$$ PV=nRT. $$This is the ideal gas law. It does not describe a particular gas, but represents an abstract model, built by removing some details of reality in order to focus on the fundamental relations between pressure, volume, temperature and amount of substance.
As long as these simplifications are reasonable, the model works very well. But what happens when the piston compresses the gas a great deal?
At that point the molecules come closer together. Their volume can no longer be ignored and the forces they exert on one another become important. Molecules, in fact, are not mathematically described points and are not completely independent. They attract and repel one another.
The real gas is therefore different from the ideal gas precisely because reality introduces constraints that in the ideal model one chooses to ignore.
This is where Johannes Diderik van der Waals comes in, a Dutch physicist who lived between 1837 and 1923. In 1873 he proposed an equation capable of correcting the ideal gas law by introducing precisely the effects due to the volume of the molecules and to their interactions:
$$ \left(P+\frac{an^2}{V^2}\right)(V-nb)=nRT. $$The two new terms have a very concrete physical meaning.
The term $b$ accounts for the space the molecules take away. If the cylinder has volume $V$, not all of this space is really available for the movement of the particles, because part of it is occupied by the molecules themselves and part is made inaccessible by the fact that two molecules cannot come closer than a certain limit. This is why $V-nb$ appears in the equation, where $nb$ is the so-called excluded volume.
The term $\frac{an^2}{V^2}$ represents instead the effect of the attractions between the molecules. A molecule about to strike the piston is pulled back by the companions it has left behind, and therefore arrives less violently than it would if it were truly free. The result is that the pressure actually measured on the piston is lower than the one the ideal model would predict, and this is why the term has to be added to $P$ in order to reconstruct the ideal value.
On the left the ideal gas, whose molecules are treated as points with no volume and with no interactions between them. On the right the real gas, where the dashed circle around every molecule marks out the space the others cannot occupy, while the lines indicate the forces of attraction.
Van der Waals therefore makes an important move: he does not abandon the ideal model, he corrects it by introducing the constraints of reality into its structure.
And it is precisely this move that allows us to establish a connection with regularization in the training of models.
When we train a machine learning model, we can imagine having a very free model, capable of fitting the training data. This freedom is useful, but it can become a problem. If the model is too free, it can learn not only the structure we are interested in, but also the noise and the accidental peculiarities present in the data.
Regularization then introduces a constraint. For example, the L2 penalty makes solutions characterized by overly large weights costly. The model remains free to learn, but it is no longer completely free to fit any detail whatsoever of the data.
The parallel with van der Waals is not a mathematical identity. His equation is not an L2 regularization and the term $\frac{an^2}{V^2}$ is not, in the technical sense, the equivalent of a penalty on the weights. The similarity lies instead in the logic of modeling.
In the ideal gas we start from a simple model free of constraints, and in the real gas we introduce information about the physical structure of the system, in that molecules occupy space and interact with one another. In machine learning we start from a model that has a certain capacity to fit the data, and with regularization we introduce a constraint that limits this freedom and keeps it from chasing indiscriminately whatever it observes in the training data.
In both cases, then, the fundamental move is the same: a model that is too free is made more structured through the introduction of constraints.
The piston offers us a good image for grasping this idea. In the ideal gas the piston can be described through a simple relation, because we have chosen to ignore what happens at the microscopic level. In the real gas, on the other hand, the movement of the piston is affected by the concrete presence of the molecules, by their volume and by their interactions. In the same way, in regularization we do not ask the model to stop learning, we impose on it a structure that makes its learning more controlled.
And here the circle of the three parts closes as well. In Part 1 we learned to read the error by separating it into bias, variance and noise. In Part 2 we discovered that this reading changes its nature when the problem changes, and that the very sign of the variance depends on the context. In this third part we have stopped reading and started pushing, discovering that pushing almost always means the same thing: trading a little variance for a little bias, and doing it in the right direction.
We might say that van der Waals shows us a physical version of the same insight.
In order to describe reality better, we do not always have to make the model freer. Often we have to add the right constraints.
Key terms
- Regularization: a set of techniques that constrain the model’s freedom during training to reduce overfitting and improve its ability to generalize.
- Curse of dimensionality: the phenomenon by which, as the number of variables increases, the space of configurations grows exponentially while the available data remain scattered across a tiny portion of it.
- Explicit regularization: a term added directly to the loss to penalize unwanted parameter configurations.
- Implicit regularization: a regularizing effect that emerges from practical choices such as using small batches or a particular parameter initialization, without any additional term being written into the loss.
- L2 regularization (weight decay): a form of explicit regularization that penalizes large parameters by adding the sum of their squares to the loss. It is equivalent to assuming a priori that the parameters follow a Gaussian distribution centered at zero.
- Maximum likelihood: an estimation criterion that selects the parameters that maximize the probability of observing the data. The most common loss functions derive from this criterion.
- Maximum a posteriori (MAP): an extension of maximum likelihood that incorporates a prior distribution over the parameters. The regularization term corresponds to the negative logarithm of this distribution.
- SGD (Stochastic Gradient Descent): an optimization method that updates the parameters using at each step only a random subset of the data. It introduces noise into the descent path, favoring wider minima that tend to generalize better.
- Early stopping: a technique that halts training before convergence, at the point where the error on the validation data stops improving. It prevents the model from fitting the noise in the training data.
- Implicit bias of the optimizer: the tendency of the optimization algorithm to select, among the infinitely many solutions compatible with the data, those with the smallest norm, even in the absence of an explicit regularization term.