CLIP: Architecture for Image-Text Alignment

In the previous post, we looked at how Multimodal spaces are useful for retrieval and classification. In this we discuss CLIP (Contrastive Language-Image Pre-training), a influencial model that creates such multimodal space.

• CLIP: The Dual-Encoder Architecture
• The Projection Head to address Dimensionality Mismatch.
• InfoNCE Loss to address Semantic Disconnect
• WebImageText Dataset to address Scaling.

---

Released by OpenAI in 2021, CLIP provides the mechanism to construct a robust Image-Text Multi-modal space.

CLIP: The Dual-Encoder Architecture

CLIP consists of two separate encoders (neural networks), one for image and another for text:

• The Image Encoder: Typically a Vision Transformer (ViT) or a CNN based architecture like ResNet. Given an image, the image encoder produce a high-dimensional vector.
• The Text Encoder: In general is a Transformer that maps sentences to vectors in the same dimensional space.

To address the three challenges outlined on the previous post: Dimensionality Mismatch, Semantic Disconnect, and Scaling

The Projection Head to address Dimensionality Mismatch.

Because these encoders use different architectures, their raw outputs (hidden states) rarely share the same dimensionality. For instance, a ViT might produce an embedding vector $x \in \mathbb{R}^{768}$, while a text encoder might output $y \in \mathbb{R}^{512}$.

CLIP resolves this by adding a linear projection head to cast both into a shared 512-dimensional latent space. Specifically:

• The Vision Projection is a matrix $W_v \in \mathbb{R}^{768 \times 512}$.
• The Text Projection is a matrix $W_t \in \mathbb{R}^{512 \times 512}$.

When the raw features are multiplied by their respective projection matrices, they produce two vectors in the same Euclidean space, finally allowing for direct mathematical comparison.

InfoNCE Loss to address Semantic Disconnect

The image and text features are now in the same Eucledian space. But they are not aligned, i.e. the image feature produce by the vision encoder for an image of a dog may not lie close to the text feature generated by the text encoder for the text “a photo of dog”.

CLIP utilizes a InfoNCE loss that pulls matching text and image feature together and pushes unmathing pairs away. The idea is to compute a similarity metric (distance), and reduce it for corresponding parirs, and increase them for unmatching pairs. InfoNCE compute the the cosine distance (dot-product) as a metric for alignment.

CLIP utilizes a dual loss that reduce the distance of text feature from image and vice versa. The loss is shown below.

\[\mathcal{L}_{i,j} = \underbrace{ -\log \left( \underbrace{ \frac{ \exp \overbrace{ \left( \text{sim}(x_i, y_j) / \tau \right) }^{\text{Scaled Similarity}} }{ \sum_{k=1}^N \exp(\text{sim}(x_i, y_k) / \tau) } }_{\text{Probability (Softmax)}} \right) }_{\text{Negative Log-Likelihood}}\]

We calculate the similarity (dot product) between an image and its matching text, then use a softmax to turn those scores into a probability of the model picking the correct pair out of the batch. The Negative Log-Likelihood then penalizes the model based on that probability—the less certain the model is about the correct match, the higher the loss.

WebImageText Dataset to address Scaling.

OpenAI scraped the web for images and their naturally occurring captions to create the WIT (WebImageText) dataset, which consists of 400 million image-text pairs.

CLIP was trained using a massive amount of compute to handle this scale. Specifically, it was trained on 256 V100 GPUs for 12 days. For the larger ViT-L/14 model, the training required even more resources, utilizing a batch size of 32,768 to ensure the InfoNCE loss had enough negative samples to be effective.

Based on the TDP of V100 clusters, we can estimated that training a model would approximately consume approximately 20–30 MWh of electricity. To put that in perspective, that is roughly the same amount of energy used by an average US household over two to three year

At 2021 cloud pricing, training a single final version of CLIP would cost roughly 200k – 500k.

Given the widespread applicability of CLIP today—enabling everything from zero-shot classification to the foundation of modern Generative AI—these upfront costs are easily justified.