Since the end of Dennard scaling, single-thread processor performance has improved slowly, while the compute used to train AI models has doubled roughly every six months. This module covers the move from general-purpose processors to domain-specific hardware. It introduces Amdahl's law as a limit on acceleration, describes the properties that many AI workloads share, and defines arithmetic intensity. It then presents the roofline model, which uses arithmetic intensity to determine whether a workload is limited by compute or by memory bandwidth. The module closes by introducing the five accelerators used throughout the course.

Processors differ in when and where the next operation is selected. CPUs and GPUs select operations at runtime from an instruction stream. Many AI accelerators perform part or all of this scheduling in the compiler. This module compares instruction-driven, compiler-scheduled, and data-triggered execution. It describes the operation of a systolic array cycle by cycle and its implementation in the TPU matrix unit. The module concludes with a transformer block expressed as a dataflow graph and a comparison of the workloads suited to each execution model.

For many AI workloads, memory bandwidth and data placement limit performance more than peak compute. This module presents the memory hierarchy with measured figures from an NVIDIA H100 and traces the movement of one tensor through it. It compares hardware-managed caches with software-managed scratchpads and surveys the memory organization of all five accelerators, including the internal structure of an IBM Spyre chip. The module closes with the two phases of LLM inference, prefill and decode, and quantifies the weight traffic per generated token during small-batch decode.

Accelerator programming models range from kernel languages, which expose direct control of the hardware, to graph compilers, which provide portability across targets. This module positions CUDA, Triton, torch.compile, FX, and XLA within that range and describes the compilation path from a PyTorch model to vendor kernels. It describes the native programming model of each of the five accelerators. It then covers numeric formats such as FP8 and the conditions under which reduced precision improves performance.

This lab profiles LLM inference with Qwen 2.5 0.5B Instruct on a CPU and a GPU in Google Colab. Students build a benchmarking harness, identify the operators with the highest GPU time using PyTorch Profiler, and place the workload on a roofline for the assigned GPU. The lab concludes with the first row of the Accelerator Landscape Map. An optional Dataflow Deep Dive repeats the lab on IBM Spyre hardware for students in partnered programs.

This quiz covers all four modules of the course: why AI needs specialized hardware, control-flow and dataflow execution, memory hierarchies, and the path from a PyTorch model to an accelerator. It has 16 questions, including calculations with Amdahl's law, arithmetic intensity and the roofline model. Allow about 40 minutes and keep a calculator nearby. You need 80% to pass, and you can take the quiz up to 3 times. Your highest score counts.