We're sorry but this app doesn't work properly without JavaScript enabled. Please enable it to continue.

This lesson's interactive features are locked, please to keep using them

Multimodal Embeddings

Click to play video

Ok, so query rewriting works, but it's not really multimodal search. We're summarizing an image into text, and as they say, "a picture is worth a thousand words"... so it's lossy compression.

If we want true multimodal semantic search, we need to map both modalities into the same vector space. Images and text are fundamentally different:

  • Images: Pixels, colors, shapes, spatial relationships
  • Text: Words, syntax, semantics, sequential patterns

But we want them to represent semantic meaning in the same space.

Training a Multimodal Model

Multimodal models learn this mapping through contrastive learning:

  1. Paired data: Images with their descriptions
    • A photo of a bear → "A brown bear in the forest"
    • Movie poster → "The Revenant movie poster"
  2. Training objective: Make matching pairs close, and non-matching pairs far apart
    • Bear image + "bear" text = high similarity
    • Bear image + "car" text = low similarity
  3. Shared space: Both images and text get encoded into the same vector dimensions

In this process, images and text are encoded separately and turned into embeddings by different encoder models: one that takes tokenized text as input, and one that takes pixel values.

The encoders are trained together so matching image/text pairs have embeddings that are as similar as possible, and non-matching pairs are as dissimilar as possible. If the encoders weren't trained together, their embeddings wouldn't line up in the same useful space.

That's what contrastive learning does: it trains models by comparing outputs against each other, rather than by predicting labels directly.

The Result

After training, both modalities live in the same space:

  • Image of a bear → [0.8, -0.2, 0.5, ...]
  • Text "bear" → [0.7, -0.1, 0.6, ...]
  • Text "car" → [-0.3, 0.9, -0.4, ...]

The bear image and "bear" text are semantically similar despite being different data types.

We aren't going to train a model from scratch in this course, but it's important to understand how this stuff works under the hood as you build search systems that use it!