← Back to Table of Contents

Chapter 25 β€” Multimodal Architectures

β€œMultimodal architectures are the connective tissue between seeing and understanding β€” bridging the pixel and the word.”

CLIP: Contrastive Language-Image Pre-training (2021)

Planned content: Dual encoder: ViT image encoder + Transformer text encoder. Contrastive training on 400M (image, text) pairs. InfoNCE loss on the [B, B] cosine similarity matrix. Zero-shot classification: compare image embedding to text embeddings of class names. OpenCLIP: open reproduction. SigLIP: sigmoid loss instead of softmax for better scaling.

πŸ“Š Planned diagram: CLIP architecture β€” image encoder [B, D] + text encoder [B, D] β†’ cosine similarity matrix [B, B] β†’ diagonal = positives, off-diagonal = negatives β†’ InfoNCE loss.

πŸ“Š Planned diagram: CLIP zero-shot classification β€” image embedding cosine similarity with N text template embeddings ["a photo of a {class}"].

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import torch
import clip

model, preprocess = clip.load("ViT-B/32")

image = preprocess(image_pil).unsqueeze(0)        # [1, 3, 224, 224]
text  = clip.tokenize(["a cat", "a dog", "a car"]) # [3, 77]

with torch.no_grad():
    image_features = model.encode_image(image)     # [1, 512]
    text_features  = model.encode_text(text)       # [3, 512]

    # Normalize and compute similarity
    image_features = image_features / image_features.norm(dim=-1, keepdim=True)
    text_features  = text_features  / text_features.norm(dim=-1, keepdim=True)
    similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)  # [1, 3]

BLIP-2: Bootstrapped VLMs (2023)

Planned content: Bridge frozen image encoder (ViT) and frozen LLM (OPT, Flan-T5) with a learnable Q-Former (Querying Transformer). Q-Former: N learnable queries [B, 32, 768] attend to image features via cross-attention. Two-stage training: image-text matching first, then image-to-text generation.

πŸ“Š Planned diagram: BLIP-2 architecture β€” frozen ViT β†’ Q-Former (32 learnable queries + cross-attention to image) β†’ query tokens [B, 32, 768] β†’ project β†’ prefix tokens for LLM β†’ text generation.

LLaVA: Language-Vision Assistant (2023)

Planned content: Simple approach: project ViT features to LLM token space with a linear (or MLP) projection. LLaVA-1: linear projection. LLaVA-1.5: MLP projection + higher resolution. LLaVA-NeXT (1.6): AnyRes for high-resolution image handling. Instruction fine-tuning on visual instruction data.

πŸ“Š Planned diagram: LLaVA architecture β€” CLIP ViT [B, 256, 1024] β†’ projection MLP β†’ visual tokens [B, 256, 4096] β†’ prepend to text tokens β†’ LLM.

Grounding DINO: Open-Set Object Detection (2023)

Planned content: Extend DINO (detection transformer) with language conditioning. Text encoder (BERT) + image encoder + fusion transformer β†’ detect any class described in text. Open-vocabulary detection: zero-shot detection of any noun. Combination with SAM for β€œgrounded segment anything”.

πŸ“Š Planned diagram: Grounding DINO β€” image + text β†’ dual encoder β†’ feature fusion β†’ decoder β†’ grounded boxes for text-described objects.

SAM 2: Segment Anything in Images and Video (2024)

Planned content: Streaming memory encoder for temporal consistency in video. Hiera image encoder. Memory bank: object pointers from past frames. Memory attention: cross-attend to past frames. Real-time interactive video segmentation. SA-V dataset: 51K videos.

πŸ“Š Planned diagram: SAM 2 memory architecture β€” current frame encoder + memory from t-1, t-2 β†’ memory cross-attention β†’ mask decoder.

Flamingo and Idefics (Few-Shot VLMs)

Planned content: Flamingo: interleaved image-text in context. Perceiver Resampler compresses image features. Cross-attention layers inserted into frozen LLM. Few-shot VQA via in-context examples. Idefics: open reproduction of Flamingo.

InternVL: International Large Vision Model

Planned content: Scaling ViT to ViT-6B. Joint training with CLIP-style and generative objectives. InternVL-Chat: multimodal conversation. Strong on multilingual benchmarks.

Next: Chapter 26 β€” Optimization Overview β†’


Last updated: May 2026