Classic templates have a couple of challenges. ValueProvider support for Beam I/O transforms, and lack of support for dynamic DAG. Let us check each of them in detail. Pipeline options, or compile-time parameters. To make these options available to users while launching templates, pipeline options have to be converted to runtime parameters. The ValueProvider interface, allows the templates to accept runtime parameter values, provided by end users. ValueProvider interface support, has been added for several Google and non Google I/Os. But there are few open source I/Os that lack ValueProvider support, which means, that pipelines developed using these I/Os, cannot be converted to classic templates. In classic templates, the pipeline graph gets built when the developer converts the pipeline into a template. Due to this, the shape of the graph cannot be changed based on user provided options. An example would be, if you develop the pipeline that consumes events from Google Cloud pub/sub, and would like to load this events, either there to BigQuery, or Google Cloud Storage, depending on the user's choice at runtime. With classic templates, since the dynamics selection of sync node, at template launch time is not possible, you need to create two separate variations of this template, one for writing to BigQuery, and the other for Cloud storage. To address these challenges, Google has build the next generation of templates, referred to as flex templates. With flex templates, the pipeline developer packages the pipeline artifacts into Docker image, and stages the image on Google Container Registry. In addition, the developer creates a metadata specification file on Google Cloud Storage. Users can launch a template, referring to your metadata spec file, stored on Google Cloud Storage, by passing appropriate parameter values. Behind the scenes, the template launches surveys, reads the metadata spec file, downloads the Docker image, and invokes the pipeline, using user supplied values. It's important to note that, with flex templates, the job graph is generated, when the end user launches the templates. Whereas with classic templates, the graph is generated, when the templates are created. This distinction makes flex templates more flexible than classic templates.
