Chapter 14 β OCR and Document Understanding
βOCR transforms images of language into machine-readable text β bridging vision and NLP.β
Text in Images: Problem Space
Planned content: Scene text (arbitrary orientation, perspective, fonts) vs. document text (structured, constrained). Text detection β text recognition β information extraction pipeline. Challenges: curved text, low resolution, complex backgrounds, multilingual, handwriting.
π Planned diagram: OCR pipeline β image β text detection (bounding polygons) β perspective rectification β text recognition (character sequence) β structured output.
Text Detection
Planned content: EAST: FCN predicting per-pixel text score + bounding box. DB (Differentiable Binarization): predict probability map + threshold map β binary map for text regions. PSENet, PAN: progressive scale expansion. PANNet. Polygon vs. quadrilateral detection. NMS for text proposals.
π Planned diagram: DB text detection β probability map + threshold map β binary mask β contour-based polygon extraction. Differentiable binarization formula.
Text Recognition: CRNN + CTC
Planned content: Word-level recognition. Input: rectified text crop
[1, 1, 32, W]. CNN features β[1, C, 1, W/4]β squeeze β[W/4, 1, C]β BiLSTM β[W/4, 1, num_chars]β CTC decode. CTC (Connectionist Temporal Classification) loss: allows aligning variable-length sequences without explicit segmentation. Greedy CTC decoding, beam search CTC decoding.
π Planned diagram (flowchart): CRNN architecture β CNN column features β LSTM sequence β CTC output. CTC alignment paths visualization showing many-to-one mapping.
\[\mathcal{L}_{CTC} = -\log p(y \mid x) = -\log \sum_{\pi \in \mathcal{B}^{-1}(y)} p(\pi \mid x)\]π Planned diagram: CTC decoding β showing valid paths in the lattice that collapse to the target string, blank character role.
Attention-Based Recognition (ASTER, ABINet)
Planned content: Attention mechanism for recognizing text without CTC. Rectification + sequence attention. Two-dimensional attention. Language model branch (ABINet): iterative correction using a language model on the recognized sequence.
End-to-End OCR Systems
Planned content: PGNet, ABCNet, MANGO: detect and recognize jointly in one forward pass. End-to-end training. Masked-attention for arbitrary-shape text.
Document Understanding
Planned content: Beyond raw text: understanding layout (tables, headings, columns). LayoutLM (text + layout position + vision). LayoutLMv2, LayoutLMv3 (unified text-layout-vision). Donut (Document Understanding Transformer): end-to-end, no OCR engine needed. Table structure recognition (TATR).
π Planned diagram: LayoutLMv3 input β tokens with 2D position embeddings and vision patches, all jointly encoded.
Evaluation
Planned content: Text detection: Hmean (harmonic mean of precision/recall with IoU > 0.5). Text recognition: word accuracy, edit distance, sequence accuracy. End-to-end: ICDAR competition metrics. Document: F1 on key-value extraction, table structure accuracy.
Next: Chapter 15 β Medical Imaging β
Last updated: May 2026