Introduction to Data Engineering on Google Cloud

Instructor: Google Cloud Training

Access provided by Mohammed Bin Rashid School of Government

8 modules

Gain insight into a topic and learn the fundamentals.

Beginner level

No prior experience required

9 hours to complete

Flexible schedule

Learn at your own pace

8 modules

Gain insight into a topic and learn the fundamentals.

Beginner level

No prior experience required

9 hours to complete

Flexible schedule

Learn at your own pace

What you'll learn

Understand the role of a data engineer.
Identify data engineering tasks and core components used on Google Cloud.
Understand how to create and deploy data pipelines of varying patterns on Google Cloud.
Identify and utilize various automation techniques on Google Cloud.

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

6 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

There are 8 modules in this course

In this course, you learn about data engineering on Google Cloud, the roles and responsibilities of data engineers, and how those map to offerings provided by Google Cloud. You also learn about ways to address data engineering challenges.

This section welcomes you to the Introduction to Data Engineering on Google Cloud course, and provides an overview of the course structure and goals.

What's included

1 video

This module provides an introduction to the role of a data engineer. It covers key concepts such as data sources and sinks, data formats, storage options on Google Cloud, metadata management, and the use of Analytics Hub for data sharing within and outside an organization.

What's included

8 videos1 assignment1 app item1 plugin

8 videos Total 12 minutes

Module Introduction 1 minute
The Role of a Data Engineer 1 minute
Data Sources Versus Data Sinks 2 minutes
Data Formats 1 minute
Storage Solution Options on Google Cloud 5 minutes
Metadata Management Options on Google Cloud 1 minute
Sharing Datasets using Analytics Hub 1 minute
Lab Intro: Loading Data into BigQuery 0 minutes

1 assignment Total 15 minutes

Quiz - Data Engineering Tasks and Components 15 minutes

1 app item Total 90 minutes

Lab: Loading Data into BigQuery 90 minutes

1 plugin Total 15 minutes

Accessing and completing labs 15 minutes

This module provides an overview of data replication and migration on Google Cloud. It covers the basic architecture, the 'gcloud' command-line tool, Storage Transfer Service, Transfer Appliance, and Datastream, along with their functionalities and use cases.

What's included

6 videos1 assignment1 app item

6 videos Total 7 minutes

Module Introduction 0 minutes
Replication and Migration Architecture 2 minutes
The gcloud Command Line Tool 0 minutes
Moving Datasets 1 minute
Datastream 4 minutes
Lab Intro: Datastream: PostgreSQL Replication to BigQuery 0 minutes

1 assignment Total 15 minutes

Quiz - Data Replication and Migration 15 minutes

1 app item Total 45 minutes

Lab: Datastream: PostgreSQL Replication to BigQuery 45 minutes

This module focuses on data extraction and loading processes on Google Cloud, particularly with BigQuery. It covers the basic extraction and loading architecture, the bq command-line tool, BigQuery Data Transfer Service, and BigLake as an alternative to traditional extract-load patterns.

What's included

6 videos1 assignment1 app item

6 videos Total 8 minutes

Module Introduction 0 minutes
Extract and Load Architecture 2 minutes
The bq Command Line Tool 1 minute
BigQuery Data Transfer Service 1 minute
BigLake 5 minutes
Lab Intro: BigLake: Qwik Start 0 minutes

1 assignment Total 15 minutes

Quiz: The Extract and Load Data Pipeline Pattern 15 minutes

1 app item Total 45 minutes

Lab: BigLake: Qwik Start 45 minutes

This module provides an overview of ELT (extract, load, transform) processes on Google Cloud. It covers the basic ELT architecture, a common ELT pipeline example, BigQuery's capabilities for scripting and scheduling SQL, and the functionality and use cases of Dataform.

What's included

5 videos1 assignment1 app item

5 videos Total 10 minutes

Module Introduction 0 minutes
Extract, Load, and Transform (ELT) Architecture 1 minute
SQL Scripting and Scheduling with BigQuery 3 minutes
Dataform 6 minutes
Lab Intro: Create and Execute a SQL Workflow in Dataform 0 minutes

1 assignment Total 15 minutes

Quiz - The Extract, Load, and Transform Data Pipeline Pattern 15 minutes

1 app item Total 60 minutes

Lab: Create and Execute a SQL Workflow in Dataform 60 minutes

This module provides an overview of ETL (extract, transform, load) processes on Google Cloud. It covers the basic ETL architecture, GUI tools, batch and streaming data processing options (Dataproc, Dataproc Serverless), and the role of Bigtable in data pipelines.

What's included

8 videos1 assignment2 app items

8 videos Total 13 minutes

Module Introduction 1 minute
Extract, Transform, and Load (ETL) Architecture 1 minute
Google Cloud GUI Tools for ETL Data Pipelines 2 minutes
Batch Data Processing Using Dataproc 5 minutes
Lab Intro: Use Serverless for Apache Spark to Load BigQuery 0 minutes
Streaming Data Processing Options 3 minutes
Bigtable and Data Pipelines 1 minute
Lab Intro: Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow 0 minutes

1 assignment Total 15 minutes

Quiz - The Extract, Transform, and Load Data Pipeline Pattern 15 minutes

2 app items Total 105 minutes

Lab: Use Serverless for Apache Spark to Load BigQuery 45 minutes
Lab: Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow 60 minutes

This module focuses on automation patterns and options for pipelines on Google Cloud. It covers various tools and services like Cloud Scheduler, Workflows, Cloud Composer, Cloud Run functions, and Eventarc, along with their functionalities and use cases for automation.