This module develops value-based deep reinforcement learning as bootstrapped regression. In the previous module, we replaced tabular value functions by parameterized functions: Vθ(s), Qθ(s, a), πθ(a | s). We also saw that function approximation changes the learning problem: values are no longer stored independently, targets can move as parameters change, and bootstrapped updates can become unstable. This module applies these ideas to deep action-value learning. We begin with fitted value iteration, which turns Bellman updates into regression problems. We then study Deep Q-Networks, or DQN, where a neural network represents Qθ(s, a). DQN combines Q-learning targets with two important stabilizers: replay buffers and target networks. Finally, we study common DQN variants: Double DQN, dueling networks, and prioritized replay. The goal is to understand DQN not as a mysterious deep-learning recipe, but as Q-learning plus function approximation, bootstrapped targets, replay, and stabilization.

This module introduces policy-gradient methods, a family of reinforcement-learning algorithms that optimize a parameterized policy directly rather than deriving behavior from a learned value function. Starting from the motivation for direct policy learning, the module develops the policy-gradient objective, the score-function trick that makes this objective differentiable from sampled experience, and REINFORCE, the foundational Monte Carlo policy-gradient algorithm. The module then introduces baselines as a practical variance-reduction technique and closes by motivating actor-critic methods as the natural next step once a learned baseline is introduced.

REINFORCE updates the policy directly from sampled Monte Carlo returns, but those returns are noisy — the same policy can produce wildly different outcomes from episode to episode. This module introduces actor–critic methods, which tame that variance by learning a second component, the critic, that estimates how good a state or action is and feeds that estimate back into the policy update as a baseline. Learners will see how subtracting a learned value function from the return produces an advantage signal, how that signal generalizes from the one-step TD error to the multi-step Generalized Advantage Estimator, and how actor and critic are jointly trained via separate policy and value losses. The module closes by tracing the conceptual line from basic actor–critic methods to PPO, motivating why controlling the size of policy updates matters for stable learning.

This module surveys modern deep reinforcement learning algorithms through the lens of stability, exploration, and continuous control. In the previous module, we studied policy-gradient and actor–critic methods. Vanilla policy-gradient updates can be brittle. If the policy changes too much after one update, the new policy may perform much worse than the old one, and the data collected under the old policy may no longer be reliable for updating the new one. This module studies three major algorithmic ideas. First, we study conservative policy updates through TRPO and PPO. The main idea is to improve the policy while preventing overly large policy changes. PPO implements this idea using a simple clipped surrogate objective. Second, we study DDPG, a deterministic actor–critic method for continuous-control problems. Third, we study SAC, an entropy-regularized actor–critic method that encourages exploration and often improves robustness.

This module turns deep reinforcement learning algorithms into implementation patterns. Earlier modules introduced the main algorithmic ideas: function approximation, DQN, policy gradients, actor–critic methods, PPO, DDPG, and SAC. This module asks how those ideas become working code. A deep RL implementation is not just a neural-network training loop. In supervised learning, the data are usually given in a fixed dataset. In reinforcement learning, the data are generated by an agent interacting with an environment. This means the implementation must manage environment interaction, exploration, neural-network models, optimizers, replay buffers or trajectory buffers, target networks, logging, evaluation, and reproducibility.