How do models see images and hear sounds?
Understand multimodal models through numerical representations and encoders. Distinguish image understanding, reading text, transcription, and audio understanding, and learn to check recognition results.
So far, we have discussed text: you enter text, and the model generates text. But some AI products now let you upload screenshots, photos, or recordings and ask questions about them.
You may wonder: text passes through a tokenizer and becomes numbers a model can process. How does a photo or a sound recording enter the model?
We will start with the same idea: turn different forms of information into numerical representations that a model can calculate with.
1. What does multimodal mean?
A modality is a form of information. Text, images, and audio are different modalities. A multimodal model can handle more than one form. For example, it can receive an image and text together, then generate a text answer.
| Feature you see | Ability involved |
|---|---|
| Upload a photo and ask what is in it | Image input and text output |
| Upload a recording and summarize the speech | Speech recognition, or audio understanding and text output |
| Enter text and have the product read it aloud | Speech generation |
| Upload a photo and ask for another image | Image understanding, plus image generation or editing |
One model may provide these abilities, or several models in a product may work together. For example, a product can offer voice input by first transcribing a recording, then sending that text to a language model.
A voice-input feature alone does not prove that the underlying language model can process sound directly.
2. How does a model see an image?
Suppose you upload a screenshot and ask, “What does the button at the bottom right say?” The process has three steps:
Step 1: Process the image and divide it into suitable regions.
First, the model divides the screenshot into smaller patches. It also records information about their positions, such as the first patch being at the top left and the second at the top right. These patches are an analogy to help explain the process; not every model divides images in the same way.
Step 2: Use a visual encoder to turn the image into numerical representations.
Like a tokenizer for text, a visual encoder converts colors, edges, shapes, and other visual information into numerical features the model can understand.
Step 3: Continue predicting what comes next.
After receiving the image and tokenized text data, the model again predicts the next likely token.
3. How does a model hear sound?
Like an image, sound can pass through an encoder and become groups of values that a model can process.
But common voice features in products may follow different paths.
Path 1: Transcribe speech, then let the language model answer
This process is straightforward. A specialized speech-to-text technology or model converts the spoken words into text, then sends that text to the large language model.
Here, the language model receives only the transcript. Other information can be lost, such as background sounds, pauses, tone, or changes in emotion.
Path 2: Let the model receive audio features
Another approach passes the audio through an encoder, then sends its features and a written instruction to the model. The model can process speech and may also process background sounds or music, depending on its abilities.
Receiving audio features does not automatically mean that a model can generate speech. Different models may handle understanding sound and producing sound.