Data pipelines typically fall under one of the Extra-Load, Extract-Load-Transform or Extract-Transform-Load paradigms. This course describes which paradigm should be used and when for batch data. Furthermore, this course covers several technologies on Google Cloud Platform for data transformation including BigQuery, executing Spark on Cloud Dataproc, pipeline graphs in Cloud Data Fusion and serverless data processing with Cloud Dataflow. Learners will get hands-on experience building data pipeline components on Google Cloud Platform using QwikLabs.
Offered By
Google CloudAbout this Course
86,948 recent views
Shareable Certificate
Earn a Certificate upon completion
100% online
Start instantly and learn at your own schedule.
Flexible deadlines
Reset deadlines in accordance to your schedule.
Intermediate Level
Approx. 13 hours to complete
English
Subtitles: French, Portuguese (Brazilian), Russian, English, Spanish
Shareable Certificate
Earn a Certificate upon completion
100% online
Start instantly and learn at your own schedule.
Flexible deadlines
Reset deadlines in accordance to your schedule.
Intermediate Level
Approx. 13 hours to complete
English
Subtitles: French, Portuguese (Brazilian), Russian, English, Spanish
Offered by
Google Cloud
We help millions of organizations empower their employees, serve their customers, and build what’s next for their businesses with innovative technology created in—and for—the cloud. Our products are engineered for security, reliability, and scalability, running the full stack from infrastructure to applications to devices and hardware. Our teams are dedicated to helping customers apply our technologies to create success.
Syllabus - What you will learn from this course
6 minutes to complete
Introduction
In this module, we introduce the course and agenda
6 minutes to complete
2 videos (Total 6 min)
1 hour to complete
Introduction to Batch Data Pipelines
This module reviews different methods of data loading: EL, ELT and ETL and when to use what
1 hour to complete
5 videos (Total 18 min)
5 videos
Quality considerations1m
How to carry out operations in BigQuery3m
Shortcomings3m
ETL to solve data quality issues4m
1 practice exercise
EL, ELT, ETL30m
2 hours to complete
Executing Spark on Cloud Dataproc
This module shows how to run Hadoop on Cloud Dataproc, how to leverage GCS, and how to optimize your Dataproc jobs.
2 hours to complete
9 videos (Total 49 min)
9 videos
Running Hadoop on Cloud Dataproc10m
GCS instead of HDFS6m
Optimizing Dataproc5m
Optimizing Dataproc Storage9m
Optimizing Dataproc Templates and Autoscaling4m
Optimizing Dataproc Monitoring3m
Lab Intro: Running Apache Spark jobs on Cloud Dataproc27s
Summary31s
1 practice exercise
Executing Spark on Cloud Dataproc4m
3 hours to complete
Manage Data Pipelines with Cloud Data Fusion and Cloud Composer
This module shows how to manage data pipelines with Cloud Data Fusion and Cloud Composer.
3 hours to complete
11 videos (Total 45 min)
11 videos
Components of Data Fusion2m
Building a Pipeline6m
Exploring Data using Wrangler1m
Lab: Building and executing a pipeline graph in Cloud Data Fusion17s
Orchestrating work between GCP services with Cloud Composer1m
Apache Airflow Environment1m
DAGs and Operators12m
Workflow scheduling6m
Monitoring and Logging4m
Lab: An Introduction to Cloud Composer12s
1 practice exercise
Cloud Data Fusion and Cloud Composer30m
7 hours to complete
Serverless Data Processing with Cloud Dataflow
This module covers using Cloud Dataflow to build your data processing pipelines
7 hours to complete
12 videos (Total 40 min)
12 videos
Why customers value Dataflow3m
Building Cloud Dataflow Pipelines in code3m
Key considerations with designing pipelines2m
Transforming data with PTransforms3m
Lab: Building a Simple Dataflow Pipeline17s
Aggregating with GroupByKey and Combine7m
Lab: MapReduce in Cloud Dataflow18s
Side Inputs and Windows of data4m
Lab: Practicing Pipeline Side Inputs11s
Creating and re-using Pipeline Templates3m
Cloud Dataflow SQL pipelines3m
1 practice exercise
Data Processing with Cloud Dataflow30m
4 minutes to complete
Summary
This module reviews the topics covered in this course
4 minutes to complete
1 video (Total 4 min)
1 video
Reviews
TOP REVIEWS FROM BUILDING BATCH DATA PIPELINES ON GCP
by TPFeb 1, 2020
There were some minor problem and mistake in the lab file. The python/java scripts were not explained at all. There are questions about the code itself, but then the questions were not answered.
by UBMay 27, 2020
A great course to help understand the various wonderful options Google Cloud has to offer to move on-premise Hadoop workload to Google Cloud Platform to leverage scalability of clusters.
by ADJul 16, 2020
Great course learning what it is the big advantages of using GCP for data given they have big implementations and with better performance of what it is today in on premises scenarios
by ASMay 19, 2020
Informative on various features. But cloud fusion and dataflow are not very clearly explained in detail.. expecting more on this. Want to learn more on the pipeline topic please.
Frequently Asked Questions
Can I preview a course before enrolling?
Yes, you can preview the first video and view the syllabus before you enroll. You must purchase the course to access content not included in the preview.
What will I get when I enroll?
Once you enroll and your session begins, you will have access to all videos and other resources, including reading items and the course discussion forum. You’ll be able to view and submit practice assessments, and complete required graded assignments to earn a grade and a Course Certificate.
When will I receive my Course Certificate?
If you complete the course successfully, your electronic Course Certificate will be added to your Accomplishments page - from there, you can print your Course Certificate or add it to your LinkedIn profile.
Why can’t I audit this course?
This course is one of a few offered on Coursera that are currently available only to learners who have paid or received financial aid, when available.
Will I earn university credit for completing the Course?
This Course doesn't carry university credit, but some universities may choose to accept Course Certificates for credit. Check with your institution to learn more. Online Degrees and Mastertrack™ Certificates on Coursera provide the opportunity to earn university credit.
More questions? Visit the Learner Help Center.
