Cloud Data Engineering

Seize the savings! Get 40% off 3 months of Coursera Plus and full access to thousands of courses.

Cloud Data Engineering

This course is part of Building Cloud Computing Solutions at Scale Specialization

Instructor: Noah Gift

15,920 already enrolled

Included with

Learn more

4 modules

Gain insight into a topic and learn the fundamentals.

92 reviews

Intermediate level

Recommended experience

4 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

4 modules

Gain insight into a topic and learn the fundamentals.

92 reviews

Intermediate level

Recommended experience

4 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

16 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your subject-matter expertise

This course is part of the Building Cloud Computing Solutions at Scale Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

There are 4 modules in this course

Welcome to the third course in the Building Cloud Computing Solutions at Scale Specialization! In this course, you will learn how to apply Data Engineering to real-world projects using the Cloud computing concepts introduced in the first two courses of this series. By the end of this course, you will be able to develop Data Engineering applications and use software development best practices to create data engineering applications. These will include continuous deployment, code quality tools, logging, instrumentation and monitoring. Finally, you will use Cloud-native technologies to tackle complex data engineering solutions.

This course is ideal for beginners as well as intermediate students interested in applying Cloud computing to data science, machine learning and data engineering. Students should have beginner level Linux and intermediate level Python skills. For your project in this course, you will build a serverless data engineering pipeline in a Cloud platform: Amazon Web Services (AWS), Azure or Google Cloud Platform (GCP).

This week, you will learn about the methodologies involved in Data Engineering. You will also learn to evaluate best practices for dealing with the end of Moore’s Law, develop distributed systems that apply software engineering best practices and evaluate best practices for implementing solutions with Big Data. You will apply these practices to build a GPU programming project using Numba and the CUDA SDK.

What's included

25 videos23 readings5 assignments4 discussion prompts3 ungraded labs

25 videos Total 144 minutes

Instructor Introduction 2 minutes
Course Introduction 2 minutes
Lab Onboarding 2 minutes
Course 3 Project Overview 1 minute
Introduction to the End of Moore's Law 2 minutes
The Problem with Concurrency in Python 2 minutes
Exploring the End of Moore's Law 3 minutes
Using CUDA and Numba 6 minutes
What is an ASIC? 2 minutes
Taking Advantage of Colab Pro 6 minutes
Exploring Colab AI 5 minutes
Introduction to Distributed Systems 2 minutes
Logging and Instrumentation Distributed Systems 1 minute
CAP Theorem 3 minutes
Amdahl's Law 3 minutes
Elasticity 3 minutes
Highly Available Nine Nine's 2 minutes
Debugging Python Code 25 minutes
Exploring Google BigQuery 13 minutes
What is Big Data? 2 minutes
Data Lakes 2 minutes
Big Data Processing 3 minutes
Introduction to AWS Data Engineering Design Principles 21 minutes
Processing Big Data with AWS 25 minutes
Transform Data with Databricks Spark SQL 6 minutes

23 readings Total 225 minutes

Getting Started and Course Gotchas 10 minutes
Specialization Project Roadmap: Course 3 10 minutes
Course Structure and Discussion Etiquette 10 minutes
Report a problem with the course 5 minutes
Key Terms 10 minutes
The End of Moore's Law 10 minutes
Numba User Guide 10 minutes
Custom Silicon 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Python Debugger 10 minutes
Python Logging 10 minutes
Implementing elasticity, high availability, and monitoring in the AWS cloud 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Introduction to Cloud Data Platforms 10 minutes
What is a Lakehouse Platform 10 minutes
Snowpark ML 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Turbocharging Python with Command Line Tools 10 minutes
Creating high-performance code 10 minutes
Lesson Reflection 10 minutes

5 assignments Total 150 minutes

Quiz-The End of Moore's Law 30 minutes
Quiz-Build Distributed Systems 30 minutes
Quiz-Understand Big Data and Data Processing Platforms 30 minutes
Quiz-High-Performance Code 30 minutes
Quiz 30 minutes

4 discussion prompts Total 40 minutes

Introductions 10 minutes
Implications of the End of Moore's Law 10 minutes
Monitoring and Logging In Distributed Systems 10 minutes
Challenges to Big Data Platforms 10 minutes

3 ungraded labs Total 180 minutes

Threads and Processes 60 minutes
Debug with Python PDB 60 minutes
Understanding Map Reduce 60 minutes

This week, you will learn what Data Engineering is and how to use software engineering best practices in Data Engineering. You will then apply this knowledge by building a command-line data processing tool.

What's included

16 videos10 readings4 assignments3 discussion prompts4 ungraded labs

16 videos Total 217 minutes

Introduction to Data Engineering 2 minutes
Data Driven Organizations 19 minutes
What is Data Engineering? 1 minute
Batch vs. Streaming vs. Events 2 minutes
Ingesting by Batch or Stream 21 minutes
Building CLI Tools with Click 34 minutes
Building Containerized Command-line Tools 13 minutes
Rust and Python 5 minutes
Python Calculator CLI 4 minutes
Caesar Cipher CLI 5 minutes
Advanced Testing with Amazon CodeGuru 15 minutes
Advanced Testing with AWS CodeBuild 30 minutes
Mapping Functions to CLI: Part 1 32 minutes
Mapping Functions to CLI: Part 2 28 minutes
AWS CodeWhisperer CLI 4 minutes
AWS CodeWhisperer SDK 4 minutes

10 readings Total 100 minutes

Key Terms 10 minutes
Python Generators 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Python and Rust 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
AWS CodeWhisperer for Data Engineering 10 minutes
Lesson Reflection 10 minutes
Extending Command-Line Data Processing Tool 10 minutes

4 assignments Total 120 minutes

Quiz-What is Data Engineering? 30 minutes
Quiz-Using Command-line tools in Rust and Python 30 minutes
Quiz-Advanced Code Testing and AI Enhanced Development Techniques 30 minutes
Quiz 30 minutes

3 discussion prompts Total 30 minutes

The Power of Events 10 minutes
Why is a Containerized CLI so Useful? 10 minutes
Using Advanced Code Analysis Tools 10 minutes

4 ungraded labs Total 240 minutes

Streaming Data with Python Generators 60 minutes
Build a Click Command-Line Tool 60 minutes
Rust CLI Tool 60 minutes
Building Command-Line Data Processing Tool 60 minutes

This week, you will learn serverless data engineering techniques and data governance best practices. You will then apply this knowledge by building a serverless Data Engineering system.

What's included

23 videos7 readings3 assignments2 discussion prompts

23 videos Total 213 minutes

Introduction to Serverless Data Engineering 1 minute
Automating Pipelines 21 minutes
What is Serverless? 1 minute
Serverless Concepts: Service Model 6 minutes
Serverless Concepts: Functions 4 minutes
Serverless Concepts: Ecosystem 7 minutes
AWS Lambda: Overview 1 minute
Introduction to AWS Cloud9 7 minutes
AWS Lambda: Event Handling 6 minutes
AWS Lambda: Hello World 12 minutes
AWS Lambda: Deploy Testing 5 minutes
AWS Lambda: Wikipedia Example 14 minutes
Build a Serverless Data Pipeline 38 minutes
Serverless Cookbook with AWS 33 minutes
Serverless Cookbook with GCP 17 minutes
Introduction to Data Governance 1 minute
What is Data Governance? 1 minute
The Principle of Least Privilege 2 minutes
Cloud Security with IAM on AWS 7 minutes
AWS Shared Security Model 8 minutes
AWS IAM Service 12 minutes
AWS Cloud Security Operations 5 minutes
Encrypt at Rest and Transit 4 minutes

7 readings Total 70 minutes

Key Terms 10 minutes
Lambda Console Gotchas 10 minutes
Using Amazon EFS for AWS Lambda 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Lesson Reflection 10 minutes
Building Computer Vision Label Trigger for S3 10 minutes

3 assignments Total 90 minutes

Quiz-Build a Serverless Data Engineering System 30 minutes
Quiz-Effective Data Governance 30 minutes
Quiz 30 minutes

2 discussion prompts Total 20 minutes

Serverless Tools that use a CLI 10 minutes
Startup Founder Wants To "Move Fast and Break Things" 10 minutes

This week, you will learn about key Data Engineering tasks including ETL, Cloud Databases and Cloud Storage. You will then apply this knowledge by building a serverless AWS lambda function that labels an image using the AWS Rekognition API.

What's included

20 videos11 readings4 assignments3 discussion prompts4 ungraded labs

20 videos Total 133 minutes

Introduction to Extract, Transform, Load (ETL) 1 minute
What is ETL? 1 minute
Ingesting and Preparing Data on AWS 20 minutes
Using Amazon Athena with AWS Glue 22 minutes
Real-World Problems in ETL 14 minutes
Introduction to Cloud Databases 7 minutes
One Size Does Not Fit All in the Cloud? 1 minute
MySQL Overview 3 minutes
MySQL from Terminal 3 minutes
Archive and Drop Database 5 minutes
Import external database Sakila 7 minutes
Modify database Sakila 5 minutes
Bash pipelines with MySQL 5 minutes
MySQL to Python Standard Library Web Server 4 minutes
Big Query with Prompt Engineering 9 minutes
Big Query Colab Pipeline 6 minutes
Introduction to Cloud Storage 1 minute
Why Cloud Storage? 1 minute
Cloud Storage Deep Dive 13 minutes
Using Amazon S3 5 minutes

11 readings Total 110 minutes

Key Terms 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Cloud Databases 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Cloud Storage Solutions 10 minutes
Lesson Reflection 10 minutes
Create a serverless Data Engineering Pipeline 10 minutes
Next Steps 10 minutes
Share your learning experience 10 minutes

4 assignments Total 120 minutes

Quiz-Effective ETL 30 minutes
Quiz-Cloud Databases 30 minutes
Quiz-Cloud Storage 30 minutes
Quiz 30 minutes

3 discussion prompts Total 30 minutes

Why spend four minutes when you can spend four days? 10 minutes
Relational Database Conundrum 10 minutes
Disaster Recovery and Amazon S3 10 minutes

4 ungraded labs Total 240 minutes

MySQL for Data Engineering 60 minutes
Rust Data Engineering Sandbox 60 minutes
Jupyter Sandbox 60 minutes
VS Code Sandbox 60 minutes

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV. Share it on social media and in your performance review.

Instructor

Instructor ratings

(26 ratings)

Noah Gift

Duke University

40 Courses 262,683 learners

Offered by

Duke University

Explore more from Cloud Computing

Status: Free Trial
DeepLearning.AI
Introduction to Data Engineering
Course
Google Cloud
Introduction to Data Engineering on Google Cloud
Course
Status: Free Trial
Whizlabs
Data Engineering in AWS
Course
Status: Free Trial
Duke University
Cloud Machine Learning Engineering and MLOps
Course

Why people choose Coursera for their career

Felipe M.

Learner since 2018

"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."

Jennifer J.

Learner since 2020

"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."

Larry W.

Learner since 2021

"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."

Chaitanya A.

"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."

Learner reviews

5 stars
52.17%
4 stars
25%
3 stars
7.60%
2 stars
9.78%
1 star
5.43%

Showing 3 of 92

Reviewed on Dec 30, 2021

Very important information and concepts was shared.

View more reviews

Open new doors with Coursera Plus

Unlimited access to 10,000+ world-class courses, hands-on projects, and job-ready certificate programs - all included in your subscription

Learn more

Advance your career with an online degree

Earn a degree from world-class universities - 100% online

Explore degrees

Join over 3,400 global companies that choose Coursera for Business

Upskill your employees to excel in the digital economy

Learn more

Frequently asked questions

To access the course materials, assignments and to earn a Certificate, you will need to purchase the Certificate experience when you enroll in a course. You can try a Free Trial instead, or apply for Financial Aid. The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.

When you enroll in the course, you get access to all of the courses in the Specialization, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile.

Yes. In select learning programs, you can apply for financial aid or a scholarship if you can’t afford the enrollment fee. If fin aid or scholarship is available for your learning program selection, you’ll find a link to apply on the description page.