Chapter 25 β Multimodal Architectures
βMultimodal architectures are the connective tissue between seeing and understanding β bridging the pixel and the word.β
CLIP: Contrastive Language-Image Pre-training (2021)
Planned content: Dual encoder: ViT image encoder + Transformer text encoder. Contrastive training on 400M (image, text) pairs. InfoNCE loss on the
[B, B]cosine similarity matrix. Zero-shot classification: compare image embedding to text embeddings of class names. OpenCLIP: open reproduction. SigLIP: sigmoid loss instead of softmax for better scaling.
π Planned diagram: CLIP architecture β image encoder
[B, D]+ text encoder[B, D]β cosine similarity matrix[B, B]β diagonal = positives, off-diagonal = negatives β InfoNCE loss.
π Planned diagram: CLIP zero-shot classification β image embedding cosine similarity with N text template embeddings
["a photo of a {class}"].
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import torch
import clip
model, preprocess = clip.load("ViT-B/32")
image = preprocess(image_pil).unsqueeze(0) # [1, 3, 224, 224]
text = clip.tokenize(["a cat", "a dog", "a car"]) # [3, 77]
with torch.no_grad():
image_features = model.encode_image(image) # [1, 512]
text_features = model.encode_text(text) # [3, 512]
# Normalize and compute similarity
image_features = image_features / image_features.norm(dim=-1, keepdim=True)
text_features = text_features / text_features.norm(dim=-1, keepdim=True)
similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1) # [1, 3]
BLIP-2: Bootstrapped VLMs (2023)
Planned content: Bridge frozen image encoder (ViT) and frozen LLM (OPT, Flan-T5) with a learnable Q-Former (Querying Transformer). Q-Former: N learnable queries
[B, 32, 768]attend to image features via cross-attention. Two-stage training: image-text matching first, then image-to-text generation.
π Planned diagram: BLIP-2 architecture β frozen ViT β Q-Former (32 learnable queries + cross-attention to image) β query tokens
[B, 32, 768]β project β prefix tokens for LLM β text generation.
LLaVA: Language-Vision Assistant (2023)
Planned content: Simple approach: project ViT features to LLM token space with a linear (or MLP) projection. LLaVA-1: linear projection. LLaVA-1.5: MLP projection + higher resolution. LLaVA-NeXT (1.6): AnyRes for high-resolution image handling. Instruction fine-tuning on visual instruction data.
π Planned diagram: LLaVA architecture β CLIP ViT
[B, 256, 1024]β projection MLP β visual tokens[B, 256, 4096]β prepend to text tokens β LLM.
Grounding DINO: Open-Set Object Detection (2023)
Planned content: Extend DINO (detection transformer) with language conditioning. Text encoder (BERT) + image encoder + fusion transformer β detect any class described in text. Open-vocabulary detection: zero-shot detection of any noun. Combination with SAM for βgrounded segment anythingβ.
π Planned diagram: Grounding DINO β image + text β dual encoder β feature fusion β decoder β grounded boxes for text-described objects.
SAM 2: Segment Anything in Images and Video (2024)
Planned content: Streaming memory encoder for temporal consistency in video. Hiera image encoder. Memory bank: object pointers from past frames. Memory attention: cross-attend to past frames. Real-time interactive video segmentation. SA-V dataset: 51K videos.
π Planned diagram: SAM 2 memory architecture β current frame encoder + memory from t-1, t-2 β memory cross-attention β mask decoder.
Flamingo and Idefics (Few-Shot VLMs)
Planned content: Flamingo: interleaved image-text in context. Perceiver Resampler compresses image features. Cross-attention layers inserted into frozen LLM. Few-shot VQA via in-context examples. Idefics: open reproduction of Flamingo.
InternVL: International Large Vision Model
Planned content: Scaling ViT to ViT-6B. Joint training with CLIP-style and generative objectives. InternVL-Chat: multimodal conversation. Strong on multilingual benchmarks.
Next: Chapter 26 β Optimization Overview β
Last updated: May 2026