It's all about CRT TVs and pirates

(How to watch an old TV and become a mighty AI pirate)

Until a few years ago, my parents’ house held what may well have been one of the last acts of rebellion against progress as it marches on, steady and relentless: our family’s first television set.

A cathode ray tube set, colour, enormous in both weight and size. Although “enormous” is still the way I saw it as a child, and I keep remembering it a good deal larger than it ever actually was.

I still remember when this monolith was bought, in the late eighties. Almost forty years of unbroken service, and it practically raised me. My father, I have no idea how, always managed to adapt it somehow.

Then came the point when, even though it still worked perfectly, there was no adapting it any more. But seeing it there, open, with my father at work on yet another attempt at rescue, gave me one last image as a parting gift.

Whether I am teaching a course or answering questions in a conversation, whenever the subject of LLMs comes up I now always start from the same place: “Picture an old cathode ray tube television”.

Connect the dots on the next token

Let us look at a cathode ray tube television at a scale other than the one our eyes are used to.

At first glance we see an image. But as we get closer to its microscopic scale, the image disappears and what remains is a set of physical phenomena: electrons, electromagnetic fields, energy and interactions.

Let us add that the electron crossing the tube belongs to quantum reality. We cannot simply think of it as a small particle travelling along a perfectly determined path, and indeed quantum mechanics forces us to describe its state through a probability distribution. When the process ends and the electron interacts with the screen, one of those possibilities shows up as an observable event, contributing to the light of a particular point.

The image we see is therefore the macroscopic outcome of an enormous number of microscopic events.

It is this relation between possibility and manifestation that, in my view, can help describe an LLM from at least two points of view.

The first is a comparison meant mostly in a figurative sense, useful for a discussion of two concepts that are too often misunderstood: probability and emergent property.

The single electron arrives at random, and the brightness of the point tends to stay stable with slight fluctuations. It is Boltzmann’s argument confirmed on a household appliance: regular behaviour arises from an enormous number of random events. I have already written about this in an article devoted to the subject (It’s all about the Entropy of Coffee).

The second point is more concrete and concerns a gap that ought to be part of how we talk about LLMs from the very beginning.

In physics there are the laws describing the individual components, then there is thermodynamics, which describes observable behaviour, and finally there is statistical mechanics, which connects the former to the latter. We can therefore distinguish three descriptive levels.

On the screen of a cathode ray tube, both levels are (or perhaps I should write “were”) directly observable on the same surface: the microscopic one, made of the individual arrivals of electrons, and the macroscopic one, made of the image that results from all of them together. In between, statistical mechanics describes collective behaviour in probabilistic terms, connecting microscopic phenomena to macroscopic quantities.

It is from exactly this perspective that the case of language models can be read. Here too we have, on one side, a description of how the individual components work and, on the other, an overall behaviour that emerges when those components interact.

We know the rules by which the model operates, but from those rules we are still unable to derive directly the properties the model displays at the overall level.

In this introduction to LLMs I want to start from precisely this point of view, because before getting into the technical detail of what a Transformer is, I think it is worth talking about how this theoretical gap opens up two widely debated questions (perhaps a little too widely, and in the wrong way).

The first concerns the explanation of behaviour. Saying that the model predicts the next word is a correct description of how the mechanism works, but on its own it does not amount to an explanation of the overall behaviour that emerges from the system. The distinction is there to challenge the idea that defining the mechanism is enough to explain the observed phenomenon.

The second concerns internal representations. Inside itself the model builds structures that are not explicitly present in the training data, and it is possible to intervene on those structures and obtain effects on the model’s behaviour that are, for the most part, predictable. This makes it possible to challenge the idea that the model merely combines words without developing any representation of what those words refer to.

On one side, then, the description of its mechanisms; on the other, the description of the behaviour that emerges from the interaction of those same mechanisms.

So, before going into the architecture of a language model, I would like to explain why I consider all the time spent on the polarised argument between “stochastic parrots” and “superhuman minds” to be time wasted.

“Nel momento in cui la mia attenzione si sposta dall’ordine regolare delle righe scritte e segue la mobile complessità che nessuna frase può contenere o esaurire, mi sento vicino a capire che dall’altro lato delle parole c’è qualcosa che cerca d’uscire dal silenzio, di significare attraverso il linguaggio, come battendo colpi su un muro di prigione.”

(Italo Calvino, “Mondo scritto e mondo non scritto”, 1983. Calvino describes the moment when attention leaves the orderly lines on the page and follows what no sentence can contain, sensing that something on the other side of the words is trying to break the silence.)

The three trials

As in the three trials of the first Monkey Island (if you have never played it, start by fixing that unforgivable gap), let us see how far along we are on our own quest to become mighty AI pirates.

The first trial: the individual components

At this level we know the operating rules well. Inside an LLM there are layers carrying out clearly defined mathematical operations, mainly multiplications between matrices of real numbers. Once the model’s parameters and the input text are fixed, these operations always produce the same results.

A doubt may well arise here, especially for anyone who has already used an LLM: if the computation is always the same, how can the model give different answers to the same question?

The answer lies in telling two steps apart (to keep things simple I will say “word” rather than “token”).

The first step is the computation inside the model. Given a certain text, the model always determines the same probability distribution over the next word. In other words, it does not necessarily select which word to use; it assigns a probability to each of the possible words. It might decide, for instance, that one word has a 40% chance of being chosen, another 20%, another 5%, and so on until all the possibilities are covered. All the probabilities add up to 1.

The second step is choosing the word. If this is done by sampling, the system draws one of the words according to the probabilities just computed. Repeat the draw and a different word may come out, even though the starting probability distribution is exactly the same. The chosen word then enters the text and the model computes a new distribution for the word after it. Repeating this process builds the whole answer.

So there is no contradiction: the model’s computation is deterministic, while generating the answer can introduce randomness at the moment the next word is chosen.

The second trial: the overall behaviour

At this level we have very solid empirical laws, but we do not know how to derive them from the components of the model.

The most important ones are the scaling laws. They describe how the loss changes as the number of parameters, the amount of data or the computation used for training increases. The loss is the measure of how much the model gets wrong (I covered it in the series of articles on neural networks).

These quantities are tied to one another by regularities that look remarkably simple. Increase the number of parameters, for instance, and the loss decreases along a curve that can be described mathematically by a power law. In practice this means we can predict fairly well how much the model will improve if we make it larger or give it more data and more computation.

These regularities have been observed across many orders of magnitude and are used in practice to decide how to size a model and how much computation or data to devote to training it.

For anyone who wants to dig deeper, these two papers are a good starting point:

• Scaling Laws for Neural Language Models(J. Kaplan et al.).
• Training Compute-Optimal Large Language Models (J. Hoffmann et al.).

It is the same situation thermodynamics was in before Boltzmann. The ideal gas law was known, verified and used to design machines many decades before anyone explained pressure starting from the collisions of molecules.

Having a reliable overall law without knowing how to derive it from the level below is a normal stage in the development of a theory.

The third trial: the connection

This is the level that is missing, and it is the one interpretability research is focused on.

A survey signed by twenty-nine researchers from eighteen organisations lists the open questions, states that the field still lacks solid theoretical foundations, and collects the cases in which an interpretation looked solid and later turned out to be wrong.

Open Problems in Mechanistic Interpretability (L. Sharkey et al.)

It should be said, though, that we are not groping about in complete darkness, but that is a subject for an article of its own.

Here I will simply mention a few observed phenomena that I find rather fascinating and still open to discussion, and that I like to bring up because they strike me as fairly close to phase transitions.

A phase transition is the process by which a substance moves from one physical state to another, for example from solid to liquid or from liquid to gas. The change happens when conditions such as temperature or pressure vary, and it involves a change in the physical properties of the substance.

During a phase transition, matter does not change its chemical composition: it remains the same substance, but the way its particles are arranged and move does change. When ice melts, for instance, the water is still H₂O, but it goes from a solid, ordered structure to a liquid one, in which the molecules can move more freely.

The appearance of induction heads. During training, the model gradually learns to predict the words missing from a text. Its prediction error, measured by the loss, decreases as the model improves. At a certain point, however, the loss curve suddenly changes course. That change is the sign of the appearance of the so called induction heads, particular attention mechanisms that let the model recognise an example it has already met in the text and use it to predict what comes next.

If the model has already met the sequence “A B” in the text, for instance, and later meets “A” again, it can learn to use the first occurrence to predict “B”. It is an elementary form of in-context learning, where the model uses the information present in the text it is reading, without having to be trained again.C. Olsson et al., 2022

Grokking. During training, a model can learn to answer correctly on the examples it is shown without having learned the general rule behind them: in other words, it can learn the training data by heart. For a long time, then, the model keeps getting new examples wrong. Then, all of a sudden, its ability to generalise can improve: the model starts to recognise the underlying rule and to use it on data it has never seen. This phenomenon, in which generalisation emerges long after the model has already learned the training data, is called grokking. A. Power et al., 2022

Emergent abilities. Some capabilities of language models seem to follow a peculiar course as the size of the model grows. The improvement looks slow at first and then, beyond a certain threshold, seems to become much faster. This behaviour has been described as an emergent ability, that is, a capability which, under certain ways of evaluating it, seems to appear once the model reaches a certain size.

The apparent sudden appearance of these capabilities, however, can also depend on how we measure them. If we evaluate the model by counting how many answers are completely correct, for instance, a gradual improvement can look like a sudden jump. The metric only considers whether an answer is right or wrong, and can therefore make partial improvements less visible. With more continuous measures, which take those improvements into account as well, the apparent jump can soften or disappear.

This is why the matter is still under discussion. Do emergent abilities reflect a real and sudden change in the model’s behaviour or are they, at least in some cases, partly an effect of the way performance is measured? J. Wei et al., 2022 R. Schaeffer, B. Miranda, S. Koyejo, 2023)

Pirates and stochastic parrots

The phenomena described so far show that, during training, mechanisms and capabilities can appear in models that are not easily traced back to simple memorisation of the data.

These phenomena lead to a more fundamental question, though. What does a model actually learn while it develops these capabilities? Observing that behaviour changes does not yet tell us what information the model has acquired, nor how that information is organised inside it. To answer the question we therefore need to look beyond the output and study the model’s internal representations.

This point matters a great deal in the debate around the so called stochastic parrot. The expression, introduced by Emily Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell in 2021, refers to the idea that a language model can produce plausible text by combining statistical regularities learned from the data without necessarily having a representation of the meaning of what it is saying. The image of the parrot is there precisely to separate the ability to reproduce linguistic forms correctly from the ability to understand what those forms refer to. Bender et al., 2021

The same distinction had already been addressed the year before in another paper, where the two authors (Emily Bender and Alexander Koller) discuss the problem of what can be learned from linguistic form alone and argue that the ability to recognise the regularities of language is not, in itself, a guarantee that its meaning has been acquired. Bender and Koller, 2020

So if we want to work out how apt this description of the models is, one way forward is to look at what is encoded in their internal representations. Language models turn inputs into numerical representations that are processed through the various layers of the network. Those representations can hold information that is not immediately visible in the text the model produces. The question then becomes whether they contain only information about linguistic regularities or also representations of what language describes.

In a language model, activations are the intermediate numerical values computed by the neural network while it processes a text.

They represent the temporary computational state of the network in response to a specific input and change dynamically during processing.

The idea of a gas introduced in the articles on neural networks is useful here. Activations can be compared to the microscopic state of a gas. Just as the position and velocity of the molecules change continuously with the conditions of the system, activations change while the prompt is being processed and describe the temporary internal state of the network.

A first useful example for tackling this question comes from a case that has nothing directly to do with language. Othello is a board game in which moves change the arrangement of the discs on the board. To predict which move to make, knowing the sequence of previous moves may therefore not be enough. It is also necessary to keep track of the state of the board, because it is the position of the discs that determines which moves are possible.

The case in question is in fact made up of two experiments, carried out by different research groups.

In the first experiment a model was trained to predict the next moves of a game of Othello, given nothing but the sequences of moves. The model received no explicit representation of the board. Even so, by analysing its activations, the researchers found a representation from which the state of the board could be reconstructed. Li et al., 2023

A second group of researchers then went back to that same model and showed that the representation has a linear structure. Nanda, Lee and Wattenberg, 2023

The result is interesting because the model receives only a sequence of moves, yet it seems to develop inside itself a representation of the state those moves describe. The experiment does not prove that the model “understands” Othello in the human sense of the word. It does show that learning from a sequence of symbols can lead to the formation of an internal representation of something that goes beyond the sequence itself. If a model trained on sequences of symbols can develop a representation of the state of a board, it is worth asking whether it can do something similar with the entities and properties described by language.

A second study went looking, in the internal representations of language models, for information about the geographical and temporal position of various entities. The results show that the activations do hold information about position in space and time, and that this information keeps a consistent structure even when the wording used to query the model changes. Gurnee and Tegmark, 2024

These results matter for the stochastic parrot discussion because geographical or temporal position is a property of the entity described, not of the word used to name it. If a structure corresponding to that property emerges in the model’s internal representations, explaining the phenomenon as a simple association between words is no longer straightforward. There seems instead to be an internal structure that preserves information about what those words describe.

One problem remains, though. The fact that a certain piece of information can be located in the internal representations does not necessarily mean the model uses it to determine the current output. A property can be present in the activations without playing any causal role in generating the output. Telling these two cases apart therefore takes more than observing the representations. It takes intervening on them and checking whether changing them produces a change in the model’s behaviour.

This is exactly the logic of the experiments that go under the name of activation steering. It has been shown that a model’s activations can be acted upon along specific directions, systematically altering certain features of the output. This kind of result provides stronger evidence than simple correlation, because it makes it possible to check whether a given internal structure actually contributes to producing the behaviour. Turner et al., 2023

The distinction is therefore between a representation that is merely associated with a behaviour and a representation that helps determine it. In the first case we can say that a certain piece of information is present in the model. In the second we have evidence that the information plays a part in how the model produces its output.

The presence of internal representations is not, on its own, enough to establish that a model has an understanding of meaning comparable to a human one. The experiments do show something important for the discussion we started from, though. If a model can develop structured representations of what sequences of symbols describe, and those representations can contribute to its behaviour, then how it works does not seem to be exhausted by the simple recombination of the linguistic forms met during training.

This is precisely the point that calls into question a reading of the model as a stochastic parrot. There is no need to conclude that models understand language the way human beings do. It is enough to observe that, at least on this evidence, between the simple statistical manipulation of words and human understanding there is a third way that would deserve some consideration. Models can develop structured internal representations of entities, properties and relations described by language, and those representations can play a part in their behaviour.

The question, then, is not only about what the model manages to produce. Having observed new mechanisms and new capabilities appear during training, we also need to ask which internal structures make those capabilities possible. It is on this ground that studies of internal representations offer an important element against the idea that the behaviour of models can be explained entirely as a simple statistical manipulation of linguistic form.

It will surely have come across in this series of articles, but I would rather say it plainly. If I am going to spend time working out how an LLM works, I would much rather travel the many roads offered by statistical mechanics.

And it is not an idea of my own, let me be clear. Statistical mechanics provides one of the deepest theoretical keys to explaining the behaviour of Large Language Models. Only, instead of looking at these models as a mysterious “biological brain”, physicists and theoretical computer scientists prefer to describe them as a complex physical system with a very high number of dimensions, governed by the statistical and thermodynamic laws of phase transition and energy minimisation.

In the articles to come I will present the literature on the subject in more detail. In the meantime, since phase transitions have come up, this work updated in June 2026 is worth mentioning at least: Kai Nakaishi, Yoshihiko Nishikawa, Koji Hukushima, 2026.

The work is interesting precisely because it applies the tools of statistical physics to an LLM explicitly. As the temperature varies, the model goes through a genuine phase transition. Below a certain threshold it produces text with repetitive structures, above that threshold it produces incomprehensible text, and at the crossing point some statistical quantities diverge, exactly as happens in physical systems.

The most significant result, however, concerns what happens near that critical point. In a text written by a person, two distant words remain connected in some way, because the topic and the structure of the discourse keep shaping the content even several lines apart. That connection weakens slowly and only as the text goes on, when the narrative context changes, for instance. And it is essentially this connection that gives language the features setting it apart from a mere sequence of symbols.

Well, the authors find that very same behaviour in the text generated by the model when the temperature is close to the critical value, and they conclude that the analogy between LLMs and natural phenomena is not just a suggestive image but something that can be measured.

So, to close, this is the line of thinking I enjoy following, at least for now, and I do not think I will be distracted from it any time soon. Not even if a three headed monkey were to turn up behind me.