How Do Models Learn?

Have you ever wondered how these foundation models become so intelligent in the first place? They aren’t born understanding language or recognizing images, right? Instead, they go through an initial phase called pretraining—AI’s equivalent of foundational education. Let’s dive deep into how this foundational education happens and why it matters.

We’ll briefly introduce the landscape of pretraining methods for modern AI and see how models like GPT rely on heavy training to understand language. First, let’s step back and explore how to train a foundation model for images, text, audio, or a combination of all three. Think of it like hiring three robot chefs to work in your restaurant kitchen:

• The first robot attended culinary school, carefully following labeled recipes with step-by-step instructions.

• The second robot never had formal instruction; instead, it studied countless cookbooks to find common cooking patterns.

• The third robot had no instructions. It experimented by cooking randomly, tasting the results, and learning what worked best.

These robots perfectly represent AI’s three main pretraining paradigms: supervised learning, unsupervised learning, and self-supervised learning. Let’s understand how exactly these models learn.

What does it mean to train a model?

When we say we’re “training a model,” we mean teaching a computer to recognize patterns from data. A model starts off knowing nothing (random parameters), and as it sees more examples, it refines its internal “brain”—the weights and biases that define its understanding.

Imagine you’re teaching a small child what a cat looks like. How would you do it? You’d probably start by showing them lots of pictures of cats. You’d point at each picture and say, “Look, this is a cat!” Initially, the child has no clue what defines a cat. They might mistakenly look at a picture of a dog and confidently shout, “Cat!” But each time this happens, you gently correct them: “No, that’s a dog—not a cat.” Eventually, after seeing enough examples, making mistakes, and getting corrected, the child learns to recognize cats on their own. They gradually notice common patterns: cats have whiskers, certain kinds of ears and tails, and usually a specific body shape. After enough practice, the child can recognize cats reliably—even ones they haven’t seen before.

Now, imagine teaching a computer to recognize cats. The process is ...