This module is the bridge into Markov Processes and Markov Chains. Thompson sampling is an old algorithm, that has been revived and is currently in-use on many challenging problems. By understanding this material and the connections to last week and to the week ahead, students will be well positioned to have mastered this first course in the specialization

Change points are locations where the previously stationary distributions of the last two modules shift to a new distribution In a manufacturing line this could be due to a new batch of materials that arrive with different characteristics, so the failure rate changes.

Markov chains describe a sequence of state changes. They are often used to describe complex transitions between states and are a primary modeling tool for improving understanding of a complex system. We will use them as a model for how sequential data may be produced by a more complex system.

The next step in modeling ability is Markov processes with decisions. This connects to modern research in reinforcement learning and enables optimization over the sets of decisions for an optimal outcome. In this last week of the first course we will cover the basics of how these Markov Decision Processes can be parameterized and what they mean.