It's all about Gas

(How to play with balloons and validate neural networks in the process.) We have just finished training our model. But the work isn’t over yet. Now we have to figure out whether the training went well. When evaluating both how the training phase unfolded and the final state the model has reached, two fundamental concepts come into play: overfitting and underfitting. In the first case the model has adapted itself excessively to the training data. In the second, on the contrary, it wasn’t able to “learn” enough. In both cases we find ourselves faced with a model incapable of generalizing. ...

If a balloon is half deflated it goes limp and holds no shape at all. We try playing with it, but no amount of squeezing gets it to take the shape we would need. A balloon that is too inflated has the opposite problem. The shape is there, but it is stretched so tight that it becomes “unstable”, to the point where the slightest squeeze risks bursting it. These are the two extremes we started from in the first part, underfitting and overfitting, looked at from the side of how well a system manages to settle. The “deflated” model has not reached any “shape” close to the one described by the data. The model that has “filled out” the shape of the data perfectly, noise included, now has no give left for taking in a new datum. ...

Given a trained model, when it gets something wrong, how do we describe the error it has made? In Part 1 we answered this question by breaking the error down into the sum of three clearly distinct parts. The bias depends on the model itself. The variance, on the other hand, depends on the fact that the model was trained on a finite sample and therefore changes as the composition of the dataset changes. And finally there is the noise, an element that is part of the data being used and cannot be removed in any way. The result we obtained from this decomposition lies precisely in its additive nature. A nature that emerges from the computation of the mean and that allows us to study these three pieces one at a time. ...