In this module, we will automate the training and tuning process with described before using this time Kubeflow pipeline. Instead of having to trigger every single step of the process manually from the Jupiter Lab Notebook, we can trigger the entire process with a single click after we have expressed the various steps as a Kubeflow Pipeline. Let's start by describing some high level concepts. It's very easy to think that machine learning code, then machine learning model development take up most of the work. In reality, most of the machine learning systems are large, complicated and distributed. And most of the effort is in the DevOps work that surrounds the actual model code development. Kubeflow was developed to use Kubernetes to standardize and streamline the DevOps work around machine learning. We know that the machine learning process can be split into sequence of more or less standard steps, and model training is just one of them. These steps can be arranged in a pipeline of tasks that are organized into a directing, basically graph. So for instance, here is a very simple and relatively standard machine learning pipeline. Is just a strain of tasks that starts with loading the data from a data store, possibly analyzing and extracting the raw data. And then a series of tasks that are linked to model development that is, model preparation, model training, model evaluation, and validation. The last task is after the model has been trained to export it so that it can be later deployed. The DevOps involved in the machine learning process can be quite difficult, why? Because during model development and experimentation, the part of the code that includes data preparation, model training, and model evaluation will change significantly. The part of the pipeline that concerns model development will change to produce different models with different performances. This model must be compared so that the best model is saved and deployed to prediction. We need an infrastructure that's flexible enough to allow these different versions of the model development pipeline to coexist in the same environment. That's a tricky DevOps problem. So for that Kubeflow provides standardized platform to build this machine learning pipelines. There are some very good aspects of the platform. It leverages containers and Kubernetes so it can be run anywhere Kubernetes is already running, especially on premises with Anthos with Google Kubernetes Engine. It's open source, it's cloud native, it's composable, portable, and standard, which means that you can create a component, reuse it in a different pipeline, and share it with your colleagues. To sum up, Kubeflow allows you to orchestrate the entire machine learning workflow. You can share and re-use the components in a very standardized way, that allows for rapid and reliable experimentation. Okay, let's look at what constitutes a Kubeflow pipeline. In a Kubeflow pipeline, each of these machine learning tasks is conceptualized as a docker container. Each task corresponds to a docker container that's executed. What kind of tasks will these containers execute? This tasks can be data import, model training, model serving, or modal evaluation like the task we saw earlier in the simple chime ML pipeline. This is useful because containers are really portable, so you can run these tasks anywhere you can run a container. Also available are prebuilt containers that allow you to use them, any resources from Google Cloud, an AI Platform such as Dataflow, Dataproc, BigQuery, AI, Platform Training, and AI Platform Prediction. The simple Python SDK allows you to describe the pipeline in code. The Kubeflow UI allows you to visualize the pipeline and view the different tasks. You can click each of these tasks for more information, such as where the corresponding container is located. In this case, the pipeline as a first data processing stage, and then a second stage which is feature engineering. In this pipeline, we decided to train various models on the same data set to see which one performed best. This is done by the next component, which looks at the model training output to score which model performed best and then deploys it. This is another example of what a machine learning pipeline that is a bit more complex than the single chain of machine learning tasks we started with. It's also interesting that after the tasks I've been executive, some artifacts are saved. These artifacts of different characteristics. For example, if you look at the artifacts for the training component, you might see the various performance matrix, the various training curves, the RC curves, etc. From the UI, you can see all the configs, and all the inputs, and outputs of each of the components. So you will never say, I have trained a model, but I can't remember which parameters I trained that model wave or which data set I used for the model. All that information is saved so you always have access to it. In terms of coding, what does it look like to implement a Kubeflow pipeline? Well, you can simply do it in the notebook if all the corresponding tasks are already available as Docker images pushed to a container registry. Start by importing the Kubeflow Python SDK, then implement the pipeline by composing the various tasks with each other, creating a graph. And then compiled by applying to a YAML pipeline description file which you upload the Kubeflow cluster. All that can be done directly in the notebook. You can also upload the pipeline description file manually through the Kubeflow UI. After you upload the pipeline, just enter the run parameters where the input is located, where the output is to be stored, which project it is, the Google Cloud region, where the pipeline will be executed, the location of the training data, the hyperparameters values, etc. Do that and then create the run, and the pipeline will run with these parameters. So that's the basics of writing and executing a Kubeflow pipeline.
