Cloud Data Engineering

Cloud Data Engineering

This course is part of Building Cloud Computing Solutions at Scale Specialization

Instructor: Noah Gift

Access provided by VodafoneZiggo

15,915 already enrolled

4 modules

Gain insight into a topic and learn the fundamentals.

92 reviews

Intermediate level

Recommended experience

4 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

4 modules

Gain insight into a topic and learn the fundamentals.

92 reviews

Intermediate level

Recommended experience

4 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

16 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your subject-matter expertise

This course is part of the Building Cloud Computing Solutions at Scale Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

There are 4 modules in this course

Welcome to the third course in the Building Cloud Computing Solutions at Scale Specialization! In this course, you will learn how to apply Data Engineering to real-world projects using the Cloud computing concepts introduced in the first two courses of this series. By the end of this course, you will be able to develop Data Engineering applications and use software development best practices to create data engineering applications. These will include continuous deployment, code quality tools, logging, instrumentation and monitoring. Finally, you will use Cloud-native technologies to tackle complex data engineering solutions.

This course is ideal for beginners as well as intermediate students interested in applying Cloud computing to data science, machine learning and data engineering. Students should have beginner level Linux and intermediate level Python skills. For your project in this course, you will build a serverless data engineering pipeline in a Cloud platform: Amazon Web Services (AWS), Azure or Google Cloud Platform (GCP).

This week, you will learn about the methodologies involved in Data Engineering. You will also learn to evaluate best practices for dealing with the end of Moore’s Law, develop distributed systems that apply software engineering best practices and evaluate best practices for implementing solutions with Big Data. You will apply these practices to build a GPU programming project using Numba and the CUDA SDK.

What's included

25 videos23 readings5 assignments4 discussion prompts3 ungraded labs

25 videos Total 144 minutes

Instructor Introduction 2 minutes
Course Introduction 2 minutes
Lab Onboarding 2 minutes
Course 3 Project Overview 1 minute
Introduction to the End of Moore's Law 2 minutes
The Problem with Concurrency in Python 2 minutes
Exploring the End of Moore's Law 3 minutes
Using CUDA and Numba 6 minutes
What is an ASIC? 2 minutes
Taking Advantage of Colab Pro 6 minutes
Exploring Colab AI 5 minutes
Introduction to Distributed Systems 2 minutes
Logging and Instrumentation Distributed Systems 1 minute
CAP Theorem 3 minutes
Amdahl's Law 3 minutes
Elasticity 3 minutes
Highly Available Nine Nine's 2 minutes
Debugging Python Code 25 minutes
Exploring Google BigQuery 13 minutes
What is Big Data? 2 minutes
Data Lakes 2 minutes
Big Data Processing 3 minutes
Introduction to AWS Data Engineering Design Principles 21 minutes
Processing Big Data with AWS 25 minutes
Transform Data with Databricks Spark SQL 6 minutes

23 readings Total 225 minutes

Getting Started and Course Gotchas 10 minutes
Specialization Project Roadmap: Course 3 10 minutes
Course Structure and Discussion Etiquette 10 minutes
Report a problem with the course 5 minutes
Key Terms 10 minutes
The End of Moore's Law 10 minutes
Numba User Guide 10 minutes
Custom Silicon 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Python Debugger 10 minutes
Python Logging 10 minutes
Implementing elasticity, high availability, and monitoring in the AWS cloud 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Introduction to Cloud Data Platforms 10 minutes
What is a Lakehouse Platform 10 minutes
Snowpark ML 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Turbocharging Python with Command Line Tools 10 minutes
Creating high-performance code 10 minutes
Lesson Reflection 10 minutes

5 assignments Total 150 minutes

Quiz 30 minutes
Quiz-The End of Moore's Law 30 minutes
Quiz-Build Distributed Systems 30 minutes
Quiz-Understand Big Data and Data Processing Platforms 30 minutes
Quiz-High-Performance Code 30 minutes

4 discussion prompts Total 40 minutes

Introductions 10 minutes
Implications of the End of Moore's Law 10 minutes
Monitoring and Logging In Distributed Systems 10 minutes
Challenges to Big Data Platforms 10 minutes

3 ungraded labs Total 180 minutes

Threads and Processes 60 minutes
Debug with Python PDB 60 minutes
Understanding Map Reduce 60 minutes

This week, you will learn what Data Engineering is and how to use software engineering best practices in Data Engineering. You will then apply this knowledge by building a command-line data processing tool.

What's included

16 videos10 readings4 assignments3 discussion prompts4 ungraded labs

16 videos Total 217 minutes

Introduction to Data Engineering 2 minutes
Data Driven Organizations 19 minutes
What is Data Engineering? 1 minute
Batch vs. Streaming vs. Events 2 minutes
Ingesting by Batch or Stream 21 minutes
Building CLI Tools with Click 34 minutes
Building Containerized Command-line Tools 13 minutes
Rust and Python 5 minutes
Python Calculator CLI 4 minutes
Caesar Cipher CLI 5 minutes
Advanced Testing with Amazon CodeGuru 15 minutes
Advanced Testing with AWS CodeBuild 30 minutes
Mapping Functions to CLI: Part 1 32 minutes
Mapping Functions to CLI: Part 2 28 minutes
AWS CodeWhisperer CLI 4 minutes
AWS CodeWhisperer SDK 4 minutes

10 readings Total 100 minutes

Key Terms 10 minutes
Python Generators 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Python and Rust 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
AWS CodeWhisperer for Data Engineering 10 minutes
Lesson Reflection 10 minutes
Extending Command-Line Data Processing Tool 10 minutes

4 assignments Total 120 minutes

Quiz 30 minutes
Quiz-What is Data Engineering? 30 minutes
Quiz-Using Command-line tools in Rust and Python 30 minutes
Quiz-Advanced Code Testing and AI Enhanced Development Techniques 30 minutes

3 discussion prompts Total 30 minutes

The Power of Events 10 minutes
Why is a Containerized CLI so Useful? 10 minutes
Using Advanced Code Analysis Tools 10 minutes

4 ungraded labs Total 240 minutes

Streaming Data with Python Generators 60 minutes
Build a Click Command-Line Tool 60 minutes
Rust CLI Tool 60 minutes
Building Command-Line Data Processing Tool 60 minutes

This week, you will learn serverless data engineering techniques and data governance best practices. You will then apply this knowledge by building a serverless Data Engineering system.

What's included

23 videos7 readings3 assignments2 discussion prompts

23 videos Total 213 minutes

Introduction to Serverless Data Engineering 1 minute
Automating Pipelines 21 minutes
What is Serverless? 1 minute
Serverless Concepts: Service Model 6 minutes
Serverless Concepts: Functions 4 minutes
Serverless Concepts: Ecosystem 7 minutes
AWS Lambda: Overview 1 minute
Introduction to AWS Cloud9 7 minutes
AWS Lambda: Event Handling 6 minutes
AWS Lambda: Hello World 12 minutes
AWS Lambda: Deploy Testing 5 minutes
AWS Lambda: Wikipedia Example 14 minutes
Build a Serverless Data Pipeline 38 minutes
Serverless Cookbook with AWS 33 minutes
Serverless Cookbook with GCP 17 minutes
Introduction to Data Governance 1 minute
What is Data Governance? 1 minute
The Principle of Least Privilege 2 minutes
Cloud Security with IAM on AWS 7 minutes
AWS Shared Security Model 8 minutes
AWS IAM Service 12 minutes
AWS Cloud Security Operations 5 minutes
Encrypt at Rest and Transit 4 minutes

7 readings Total 70 minutes

Key Terms 10 minutes
Lambda Console Gotchas 10 minutes
Using Amazon EFS for AWS Lambda 10 minutes
Lesson Reflection 10 minutes
Key Terms 10 minutes
Lesson Reflection 10 minutes
Building Computer Vision Label Trigger for S3 10 minutes

3 assignments Total 90 minutes

Quiz 30 minutes
Quiz-Build a Serverless Data Engineering System 30 minutes
Quiz-Effective Data Governance 30 minutes

2 discussion prompts Total 20 minutes

Serverless Tools that use a CLI 10 minutes
Startup Founder Wants To "Move Fast and Break Things" 10 minutes

This week, you will learn about key Data Engineering tasks including ETL, Cloud Databases and Cloud Storage. You will then apply this knowledge by building a serverless AWS lambda function that labels an image using the AWS Rekognition API.