How does a large language model generate an answer?
Understand large language models as automatic text continuation systems, and learn why your input affects their output.
You have probably seen or heard the term “large language model” recently. People often shorten it to LLM. Many call it “artificial intelligence.” Is that accurate? What is it, and why does it seem like the artificial intelligence in science fiction?
1. An automatic text continuation system
As a simple analogy, think of an LLM as an automatic text continuation system. For example, suppose you ask an LLM, such as DeepSeek or ChatGPT:
The model performs a series of calculations. But the process differs from what you might imagine. It does not “think”. It uses calculations to predict which “character” is most likely to come after this question.
For “How is the weather in Guangzhou today?”, one generation might produce “T” as the next character.
After generating “T”, the model uses “How is the weather in Guangzhou today?T” as its new input. It calculates the next character again, produces “o”, and gets “How is the weather in Guangzhou today?To”. It continues calculating the next character until it reaches the final result.
The process looks roughly like this:
In the product, you eventually see an answer like this:
Once you understand this principle, you can see why LLM products display answers one character at a time.
2. Why is this useful?
This simple principle helps explain why so many prompt engineering techniques appeared in early LLM products, such as those based on GPT-3.5.
For example, before entering English text to translate, you might add an instruction such as “You are an expert in translation between English and Chinese. Please translate the following English text…”. These instructions, or prompts, aim to increase the probability that the LLM produces a satisfactory answer.
Why?
To interact with AI, you must express what you want in writing. Without the relevant concepts, it is hard to make your requirements clear. In other words, it is hard to ask an LLM a useful question or give it suitable instructions. If those instructions are unclear or incorrect, the answer can be poor or wrong.
Try clicking each of the two buttons below. Each opens a menu with the same items. But the menus look very different. Which gives you a better experience?
If you know nothing about programming or product design, you might rely on a single sentence: “When I click the button, show a menu with three buttons: Menu A, B, and C.”
The model will probably give you the statistically most likely result, which may be ordinary. With the relevant knowledge, you can give the LLM specific instructions to improve its output.
That is what this course aims to teach you.