Chapter 39 β VisionβLanguage Models
βVision-language models donβt just see β they understand, describe, answer, and generate. They are the bridge between pixels and knowledge.β
The Vision-Language Pre-training Revolution
Planned content: History: from dual-encoder retrieval (VSE++) to contrastive pre-training at scale (CLIP). The data flywheel: internet image-text pairs at web scale. Zero-shot transfer: no task-specific fine-tuning needed. The emergent behaviors from scale.
π Planned diagram: VLM evolution timeline β VSE++ β ImageBERT β CLIP β ALIGN β BLIP β BLIP-2 β LLaVA β GPT-4V β Gemini Vision.
CLIP Ecosystem
Planned content: Original CLIP (OpenAI). OpenCLIP: open reproduction, ViT-L/14, ViT-G/14. SigLIP: sigmoid loss, better scaling. EVA-CLIP: CLIP + MAE for ViT-18B. MetaCLIP: curated training data. CLIPA: larger crops, better efficiency. Applications: zero-shot classification, image-text retrieval, image search, visual RLHF reward.
π Planned diagram: CLIP zero-shot pipeline β image features vs. class name text features, similarity-based classification without any fine-tuning.
Open-Vocabulary Detection and Segmentation
Planned content: OWL-ViT: CLIP ViT backbone + detection head, text queries. Grounding DINO: open-set detection with language grounding. GLIP: phrase grounding. GDINO-SAM: ground + segment pipeline. APE, GenerateU. These models enable detection of arbitrary object categories.
π Planned diagram: Open-vocabulary detection pipeline β text query βperson carrying a bagβ β Grounding DINO β bounding box.
GPT-4V, Gemini, and Claude Vision
Planned content: GPT-4V: multimodal GPT-4, image + text interleaved. Capabilities: OCR, chart reading, spatial reasoning, diagram understanding. Gemini 1.5 Pro: native multimodal, 1M context, video understanding. Claude-3: vision capabilities. Limitations: hallucination, counting errors, fine-grained spatial reasoning.
π Planned diagram: Capability comparison across closed-source VLMs β VQA, chart QA, OCR, code from image, video understanding.
Video-Language Models
Planned content: VideoCLIP, InternVideo2, Video-LLaVA. Challenges: temporal alignment, long video. Temporal position encoding. Frame sampling strategies. Video QA benchmarks (ActivityNet-QA, NExT-QA, MSVD-QA). EgoSchema (egocentric video understanding).
VLMs for Dense Prediction
Planned content: SegCLIP: CLIP for semantic segmentation. CLIPSeg: image + text + visual prompts β segmentation. LSeg: language-guided segmentation. NACLIP: no training zero-shot segmentation. SAM + CLIP pipeline for open-world segmentation.
Training a Custom VLM
Planned content: Data collection: image-text pairs (LAION, CC3M, CC12M). Data quality: quality filtering with CLIP score. Training infrastructure: contrastive loss, gradient checkpointing, ZeRO. Fine-tuning for downstream tasks (linear probe, zero-shot, full fine-tune). LLaVA instruction tuning recipe.
Next: Chapter 40 β Foundation Models β
Last updated: May 2026