← Back to Table of Contents

Chapter 5 — Instruction Sets (ISAs)

PyTorch’s torch.matmul eventually becomes a stream of 1s and 0s that the CPU or GPU executes directly. Between your Python line and that binary stream are several translations — but the most fundamental boundary is the Instruction Set Architecture (ISA): the contract between software and silicon that defines exactly which operations the processor promises to support, and in which binary encoding. Understanding this contract is what lets you reason about why AVX-512 gives a 2× to 8× speedup on CPU inference, why oneDNN chooses different code paths on different x86 chips, and why the GPU’s “SIMT” execution model is the natural descendant of the CPU’s “SIMD” extensions.

This chapter builds from first principles — what an instruction actually is — up through the RISC/CISC divide, the major ISAs you encounter in practice, and the vector/SIMD extensions that are the CPU-side ancestors of GPU tensor cores. It ends by connecting every concept directly to the integers and floats your models are made of.

The one-sentence version: An ISA is the hardware/software contract — the list of operations, encodings, and rules a CPU promises to obey; the microarchitecture is the (replaceable) implementation that fulfills that promise.

The ISA vs the Microarchitecture

These two terms are constantly confused. The distinction is critical.

The payoff of this split: Intel can redesign every transistor in a new processor (“microarchitecture”) without breaking any existing software, as long as it still faithfully executes x86-64 instructions (“ISA”). Conversely, AMD and Intel can both produce x86-64 chips — same ISA, completely different silicon.

For model optimization: when you compile a model with torch.compile or export to ONNX and then AOT-compile for x86, you target an ISA (including which extensions are present). The runtime can then pick the best microarchitecture-specific implementation.

Anatomy of an Instruction

An instruction is a small, fixed-format command that the processor can decode and execute in one (or a few) cycles. Every instruction has:

1. Opcode — which operation (e.g., ADD, MUL, LOAD, BRANCH)
2. Operands — what to operate on
3. Encoding — the binary layout that represents #1 and #2

Instruction Encoding

Consider a simplified 32-bit fixed-length RISC encoding (like RISC-V’s R-type):

1
2
3
4
5
6
7
8
9
 31      25 24   20 19   15 14  12 11    7 6       0
┌──────────┬───────┬───────┬──────┬───────┬─────────┐
│  funct7  │  rs2  │  rs1  │funct3│  rd   │ opcode  │
│  7 bits  │ 5 bits│ 5 bits│3 bits│ 5 bits│  7 bits │
└──────────┴───────┴───────┴──────┴───────┴─────────┘

Example: ADD x3, x1, x2
  funct7=0000000  rs2=x2(00010)  rs1=x1(00001)  funct3=000  rd=x3(00011)  opcode=0110011
  Binary: 0000000_00010_00001_000_00011_0110011

x86-64, by contrast, uses variable-length encoding (1–15 bytes per instruction), with prefixes that modify width, segment, or lock behavior. This complexity is the defining feature of CISC — power at the cost of decoder complexity.

Operand Types

Registers

Registers are the processor’s fastest storage — typically 64-bit general-purpose registers (x86-64 has 16: rax–r15; ARM64 has 31: x0–x30). Additionally:

• SIMD/vector registers: x86 ymm0–ymm15 (256-bit), zmm0–zmm31 (512-bit); ARM v0–v31 (128-bit NEON, 2048-bit SVE max)
• Status/flags register: carries results of comparisons (ZF, CF, OF, SF)
• Program counter (PC/RIP): address of the next instruction

Register access is ~1 cycle. An L1-cache miss is ~4 cycles. A DRAM access is ~200 cycles. The ISA exposes registers; the microarchitecture may rename them internally (register renaming in out-of-order cores) to far more physical registers than the ISA requires.

RISC vs CISC: Philosophy and Hardware Consequences

This is one of the most consequential design debates in computer architecture.

The Modern Twist: x86 Translates Internally

Modern x86 CPUs are not really “CISC” in execution. Since the mid-1990s (Intel Pentium Pro, 1995), they translate x86 instructions into micro-ops (µOPs) — small, RISC-like internal operations — at decode time. The out-of-order engine then executes these micro-ops.

1
2
3
4
5
6
7
x86-64 decode pipeline:

External ISA     Internal execution
─────────────    ──────────────────────────────────────
ADDQ [rsp+8], rax  →  µOP1: LOAD   tmp1 ← [rsp+8]
                   →  µOP2: ADD    rax  ← rax + tmp1
                   →  µOP3: (writeback is merged with ADD)

This means the “CISC vs RISC” distinction is today more of a software ABI and encoding story than an execution one. x86 chips execute RISC-like µOPs internally; the complexity is in the front-end decoder.

The Big ISAs Today

x86-64 (Intel / AMD)

The dominant server and desktop ISA — the machine your CUDA code usually runs the “host” side on. Originated with Intel’s 8086 (1978) and extended to 64-bit by AMD in 2003 (hence “AMD64” or “x86-64”).

Key facts for ML practitioners:

• Your CPU host code in PyTorch compiles to x86-64
• AVX-512 and VNNI/AMX (see SIMD section) give CPUs competitive INT8/BF16 throughput for inference
• Intel Core (Raptor Lake, Meteor Lake) and AMD Zen 4/5 are the current microarchitectures
• Server: Intel Sapphire Rapids/Granite Rapids, AMD EPYC Genoa/Turin

ARM (ARMv8 / ARMv9 / AArch64)

ARM Holdings designs the ISA and licenses it; dozens of companies implement their own microarchitectures. The licensing model is why you see ARM in phones (Apple A-series, Qualcomm Snapdragon), servers (AWS Graviton, Ampere Altra), and embedded (Cortex-M).

Key facts for ML:

• Apple M-series SoCs (M1–M4) use ARMv8.5/v9 with Apple’s own microarchitecture — and share memory between CPU and Neural Engine
• AWS Graviton 3/4: 64 Neoverse cores, 300 GB/s memory bandwidth — competitive for serving
• ARM’s NEON (128-bit SIMD) and SVE/SVE2 (Scalable Vector Extension — 128 to 2048-bit vectors, vector length determined at runtime) are the ML-critical extensions
• bfloat16 native support added in ARMv8.6

RISC-V (Open ISA)

RISC-V is an open, royalty-free ISA developed at UC Berkeley (2010). Anyone can implement it without paying licensing fees — a fundamental disruption to the ARM licensing model.

1
2
3
4
5
6
7
8
9
10
RISC-V standard extensions (letters in ISA string RV64GC):
  I = Base integer (mandatory)
  M = Multiply/divide
  A = Atomic operations
  F = Single-precision float
  D = Double-precision float
  C = Compressed 16-bit instructions
  V = Vector extension (RISC-V V 1.0)
  B = Bit manipulation
  Zicsr = Control/status registers

Why it matters for ML:

• Eliminates licensing fees → lowers barriers for custom AI chip startups
• Modular design — add only the extensions you need (e.g., a custom ML extension Xnn)
• Already shipping: SiFive, Alibaba T-Head (C910), StarFive JH7110, ESWIN EIC7700X
• RISC-V V (vector) extension enables portable SIMD — same story as ARM SVE

SIMD & Vector Extensions: The CPU Ancestor of GPU Parallelism

This is the section that matters most for ML practitioners working with CPU inference.

SIMD — Single Instruction, Multiple Data — means one instruction operates on a vector of values simultaneously. Instead of adding one pair of floats, add 8 or 16 at once. The hardware is literally 8 or 16 independent adders working in lockstep.

1
2
3
4
5
6
7
8
9
10
11
Scalar (without SIMD):
  FADD xmm0, xmm1      ; add 1 × f32 = 32-bit lane

SSE (128-bit):
  ADDPS xmm0, xmm1     ; add 4 × f32 = 128-bit vector

AVX (256-bit):
  VADDPS ymm0, ymm1, ymm2   ; add 8 × f32 = 256-bit vector

AVX-512 (512-bit):
  VADDPS zmm0, zmm1, zmm2   ; add 16 × f32 = 512-bit vector

One instruction, 16 additions, executed in the same cycle (roughly). This is the CPU’s answer to parallelism before you even leave a single core.

x86 SIMD History

ARM SIMD: NEON and SVE

NEON (128-bit): standard SIMD for ARMv7/v8. All modern phones and Apple Silicon support NEON. 4× FP32 or 8× INT16 or 16× INT8 per instruction.

SVE (Scalable Vector Extension): introduced in ARMv8.2, SVE registers are not a fixed width — the hardware declares a VLEN anywhere from 128 to 2048 bits. The same binary runs correctly on any SVE implementation; longer vectors give more lanes. AWS Graviton 3 uses 256-bit SVE; A64FX (Fujitsu, used in Fugaku supercomputer) uses 512-bit SVE.

1
2
3
4
SVE vector length (VLEN) determines lanes:
  VLEN=256 → 8 × FP32 lanes   (like AVX2)
  VLEN=512 → 16 × FP32 lanes  (like AVX-512)
  VLEN=2048 → 64 × FP32 lanes (future hardware)

The ML-Specific Extensions: VNNI and AMX

These are x86 ISA extensions added specifically because neural network inference is dominated by dot products on low-precision integers:

AVX-512 VNNI (Vector Neural Network Instructions, Intel Cascade Lake 2019+):

1
2
3
4
VPDPBUSD zmm0, zmm1, zmm2
  ; Computes: zmm0 += zmm1[i] (UINT8) × zmm2[i] (INT8), 4 elements at a time
  ; 512-bit registers → 64 INT8 multiply-accumulates in ONE instruction
  ; This is exactly the quantized dot product in INT8 GEMM

AMX (Advanced Matrix Extensions, Intel Sapphire Rapids 2023+): full 2D tile registers (up to 16 rows × 64 bytes each, ~1024 bytes per tile). TMUL instructions multiply two tiles — essentially a tiny matrix unit built into the CPU.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Conceptual illustration: what VNNI does per clock in Python terms
import numpy as np

def vnni_dot_product(a_uint8: np.ndarray, b_int8: np.ndarray) -> np.ndarray:
    """
    Mimics AVX-512 VNNI: accumulate 4 INT8 pairs into one INT32 accumulator.
    Real VNNI processes 64 elements (512-bit / 8-bit) per instruction.
    
    a shape: (N,)  dtype=uint8   (activations, zero-point-shifted)
    b shape: (N,)  dtype=int8    (weights)
    returns: scalar int32 accumulation
    """
    assert a_uint8.dtype == np.uint8
    assert b_int8.dtype == np.int8
    # Widen to int32 before multiply to avoid overflow, then accumulate
    return np.sum(a_uint8.astype(np.int32) * b_int8.astype(np.int32))

# The key insight: 64 INT8 multiply-adds per AVX-512 cycle
# vs 16 FP32 multiply-adds per AVX-512 cycle
# → 4× more operations per cycle for INT8 vs FP32 on the same hardware
a = np.random.randint(0, 127, 64, dtype=np.uint8)
b = np.random.randint(-64, 63, 64, dtype=np.int8)
result = vnni_dot_product(a, b)
print(f"64 INT8 dot-product result: {result} (int32 accumulator)")

Scalar vs SIMD: A Concrete Python Analogy

Here is the intuition for SIMD, expressed in NumPy terms (NumPy itself calls SIMD instructions under the hood):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import numpy as np
import time

N = 1 << 22   # 4M elements

a = np.random.rand(N).astype(np.float32)
b = np.random.rand(N).astype(np.float32)

# --- "Scalar" loop (Python for-loop; no SIMD) ---
def scalar_add(a, b):
    out = np.empty_like(a)
    for i in range(len(a)):
        out[i] = a[i] + b[i]     # one float32 add per iteration
    return out

# --- SIMD (NumPy vectorized; maps to VADDPS on x86) ---
def simd_add(a, b):
    return a + b                  # one VADDPS zmm instruction = 16 float32 adds

t0 = time.perf_counter(); scalar_add(a, b); t1 = time.perf_counter()
t2 = time.perf_counter(); simd_add(a, b);   t3 = time.perf_counter()

print(f"Scalar loop:   {(t1-t0)*1000:.1f} ms")
print(f"NumPy SIMD:    {(t3-t2)*1000:.1f} ms")
print(f"Speedup:       ~{(t1-t0)/(t3-t2):.0f}×")
# Typical result: ~100–300× speedup
# The SIMD version also benefits from cache-friendly sequential memory access

The speedup is enormous because:

1. 16× more data processed per instruction (AVX-512 vs scalar)
2. The Python loop has interpreter overhead; NumPy eliminates it
3. SIMD also enables better memory access patterns

The Same Computation: x86-64 vs ARM64

To see the ISA contrast concretely, here is a single float32 multiply-accumulate (the core of matmul) in both ISAs:

; === x86-64 (Intel syntax) ===
; Compute: result += a * b  (scalar f32)
; Assume: xmm0 = accumulator, xmm1 = a, xmm2 = b

vfmadd231ss xmm0, xmm1, xmm2    ; xmm0 += xmm1 * xmm2  (FMA: fused multiply-add)
; VFMADD231SS: 1 instruction, 1 f32 FMA, latency ~4 cycles, throughput 0.5 cycles
; (one new result every 0.5 cycles on Zen4 / Ice Lake with dual FMA units)

; AVX-512 version (16 floats at once):
vfmadd231ps zmm0, zmm1, zmm2     ; zmm0 += zmm1 * zmm2  (16 × f32 FMA)
; 16× the throughput of the scalar version in the same cycle count

; === ARM64 (AArch64) ===
; Compute: result += a * b  (scalar f32)
; Assume: s0 = accumulator, s1 = a, s2 = b

fmadd s0, s1, s2, s0             ; s0 = s1 * s2 + s0
; Clean load-store ISA: only FMADD touches registers
; No memory operands in arithmetic instructions (unlike CISC)

; NEON version (4 floats at once):
fmla v0.4s, v1.4s, v2.4s         ; v0 += v1 * v2  (4 × f32 FMA, 128-bit NEON)
; SVE version (VLEN/32 floats at once):
fmla z0.s, p0/m, z1.s, z2.s      ; z0 += z1 * z2  (predicated, all active lanes)

The computation is identical; the encoding and addressing philosophy differ. The RISC ARM version is simpler per instruction — but needs more instructions for complex memory addressing.

VLIW: An Honorable Mention

VLIW (Very Long Instruction Word) is a third philosophy: pack multiple independent operations into one wide instruction word, letting the compiler (not hardware) figure out what can run in parallel.

1
2
3
VLIW example (conceptual 128-bit wide instruction):
  [ADD R1, R2, R3] | [MUL R4, R5, R6] | [LOAD R7, [R8+4]] | [NOP]
  ^--- four slots, all execute in parallel in one cycle ---^

Hardware stays simple (no out-of-order engine needed); the compiler takes on the scheduling burden. Used in:

• Intel Itanium (IA-64): famously complex to compile for; failed in the market
• TI C6000 DSP family: successful in signal processing, video codecs
• Older AI accelerators (some vision DSPs): still appear in edge NPUs

VLIW is relevant to you because some DSPs and NPUs you’ll target for edge inference (see Chapter 18) use it, and their compilers expose this structure.

Why This Matters for Model Optimization

Checking Your ISA Extensions in Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import subprocess, platform

if platform.machine() in ("x86_64", "AMD64"):
    # Check which AVX extensions the current CPU supports
    try:
        import cpuinfo  # pip install py-cpuinfo
        info = cpuinfo.get_cpu_info()
        flags = info.get("flags", [])
        for ext in ["avx", "avx2", "avx512f", "avx512vnni", "amx_int8", "amx_bf16"]:
            status = "✓" if ext in flags else "✗"
            print(f"  {status} {ext}")
    except ImportError:
        # Fallback: read /proc/cpuinfo
        with open("/proc/cpuinfo") as f:
            cpuflags = next(l for l in f if l.startswith("flags"))
        for ext in ["avx", "avx2", "avx512f", "avx512_vnni"]:
            print(f"  {'✓' if ext in cpuflags else '✗'} {ext}")

# PyTorch also exposes this:
import torch
print(f"\ntorch.backends.cpu.get_cpu_capability(): {torch.backends.cpu.get_cpu_capability()}")
# Returns e.g. "AVX512" or "AVX2" — used internally to select kernel variants

Knowing your CPU’s SIMD capability lets you reason about whether oneDNN will use AVX-512 VNNI kernels for INT8 and how much speedup to expect from quantization on that box.

Key Takeaways

• An ISA is the hardware/software contract — the visible instruction set, encoding, and register file — while the microarchitecture is the implementation. The same ISA can be implemented by many microarchitectures (Intel vs AMD, all x86-64).
• x86-64 is CISC in encoding but translates to internal RISC-like micro-ops for execution. ARM64 and RISC-V are load-store RISC ISAs with fixed-width encodings — simpler decoders, better suited to many-core scaling.
• SIMD extensions (SSE/AVX/AVX-512, NEON/SVE) let one instruction process a vector of values — 4×, 8×, 16×, or more — per cycle. This is the CPU-level form of data parallelism, the ancestor of GPU SIMT.
• VNNI (AVX-512) adds dedicated INT8 dot-product instructions and is the reason INT8 CPU inference can be 4× faster than FP32 on modern x86 servers.
• AMX (Intel Sapphire Rapids+) brings tile-level matrix units into x86 CPUs — blurring the CPU/accelerator boundary for inference.
• RISC-V’s open ISA eliminates licensing fees, enabling a wave of custom AI chips with proprietary ML extensions.
• Every framework optimization (oneDNN, XLA, TVM, torch.compile) ultimately targets one of these ISAs and selects the widest available vector extension at runtime.

Next: Chapter 6 — Languages & Nomenclature of the Stack, where we trace the full journey from Python down to transistors — assembly, Verilog, CUDA, Triton, and every layer in between.

← Back to Table of Contents