How is a model pretrained?

We have seen how a model uses its input and parameters to calculate token probabilities. In this lesson, simple examples show how pretraining produces those parameters. You do not need to learn the training algorithms.

1. An overview of pretraining

Pretraining lets a model learn patterns in language from large amounts of text. In simple terms, it has three steps:

01Select and clean02Convert to tokens03Prediction training

2. How does pretraining work?1

The process works roughly like this.

Step 0: Initialize the parameters.

We will continue using the earlier formula: score = a × input + b.

At the start of training, a and b have not yet been learned, so we assign initial values. Each candidate has its own a and b. For example, “umbrella” starts with a = 0.50 and b = 1.10. This step only initializes the parameters. It does not yet calculate scores or probabilities from an input.

Step 1: Read some text and predict the next token.

Using these initial parameters, the model reads a passage and predicts the next token. For example, the input is “It is raining today. Remember your…”.

In this simplified example, rain becomes the input x = 1. All three candidates use the same input, but their parameters differ, so their scores differ.

Softmax then converts the scores into probabilities that add up to 100%.

Step 2: Check the correct answer.

The model checks the original text in the training data: “It is raining today. Remember your umbrella”. The correct answer is “umbrella”.

Step 3: Adjust the parameters.

After seeing the correct answer, the model adjusts its parameters.

The earlier prediction gave “raincoat” the highest probability, about 40%. But the correct answer, “umbrella”, received only about 25%. That is a substantial gap. The training program measures this gap against the correct answer, then adjusts the parameters to raise the probability of “umbrella” for this input.

For example, it changes the parameters for “umbrella” to a = 0.60 and b = 1.20, while also adjusting the other candidates. The input remains x = 1, producing these scores and probabilities:

The input has not changed. The parameters and candidate scores have. After this adjustment, the predicted probability of “umbrella” rises from about 25% to about 45%.

Here is a demonstration of the full process:

Step 4: Repeat prediction, comparison, and parameter updates.

Initialization happens only once, at the start of training. After that, the program repeatedly predicts, compares with the source text, and adjusts the parameters based on the gap. It does not reset the initial values each round.

This adjustment does not happen all at once. It must be repeated many times. The next demonstration keeps the same input so you can see how the probability of “umbrella” changes across training rounds.

After many rounds of training, the model has learned statistical patterns in language and can generate grammatical, coherent text. It can understand the meaning of text and generate text that follows grammar and logic. This gives us a base model2.

3. Why is this useful?

There is an important concept to remember: a model's training data has a time cutoff.

In other words, the model can only answer questions about information from before its training cutoff. Without other tools (简体中文), it cannot obtain information outside its training data. For example, GPT-6's training data ends in April 2026, so it cannot know what happened after April.

A second important concept is that the “correct answer” during training is not necessarily factually correct.

Earlier, we said that the model checks the correct answer. Here, that answer is the next token in the training text. The text can contain errors, bias, or fiction. Predicting the source correctly does not mean that everything learned from it is factual.

For example, if all the training data says “a week has six days”, the model will think that a week has six days.