DeepLearning.AI
Source Systems, Data Ingestion, and Pipelines
DeepLearning.AI

Source Systems, Data Ingestion, and Pipelines

Joe Reis

Instructor: Joe Reis

Top Instructor

4,579 already enrolled

Gain insight into a topic and learn the fundamentals.
4.7

(47 reviews)

Intermediate level

Recommended experience

34 hours to complete
3 weeks at 11 hours a week
Flexible schedule
Learn at your own pace
Gain insight into a topic and learn the fundamentals.
4.7

(47 reviews)

Intermediate level

Recommended experience

34 hours to complete
3 weeks at 11 hours a week
Flexible schedule
Learn at your own pace

What you'll learn

  • Gather stakeholder needs and translate them into system requirements.

  • Implement a batch and a streaming ingestion process on AWS to ingest data from various source systems.

  • Integrate aspects of security, data management, DataOps and orchestration into the data systems you build.

Details to know

Shareable certificate

Add to your LinkedIn profile

Recently updated!

September 2024

Assessments

4 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Placeholder

Build your Cloud Computing expertise

This course is part of the DeepLearning.AI Data Engineering Professional Certificate
When you enroll in this course, you'll also be enrolled in this Professional Certificate.
  • Learn new concepts from industry experts
  • Gain a foundational understanding of a subject or tool
  • Develop job-relevant skills with hands-on projects
  • Earn a shareable career certificate from DeepLearning.AI
Placeholder
Placeholder

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV

Share it on social media and in your performance review

Placeholder

There are 4 modules in this course

In lesson 1, you will explore source systems data engineers commonly interact with. Then in lesson 2, you will learn how to connect to various source systems and troubleshoot common connectivity issues.

What's included

21 videos8 readings1 assignment3 app items1 ungraded lab

This week you will dive deep into the batch and streaming ingestion patterns. You will identify use cases and considerations for each, and then build a batch and a streaming ingestion pipeline. When looking at batch ingestion, you will compare and contrast the ETL and ELT paradigms. You will also explore various AWS services for batch and streaming ingestion.

What's included

11 videos7 readings1 assignment2 app items

In the first lesson, you will explore DataOps automation practices, including applying CI/CD to both data and code, and using infrastructure as code tools like Terraform to automate the provisioning and management of your resources. Then in lesson 2, you will explore DataOps observability and monitoring practices, including using tools like Great Expectation to monitor data quality, and using Amazon CloudWatch to monitor your infrastructure.

What's included

17 videos4 readings1 assignment3 app items

This week, you will learn all about orchestrating your data pipeline tasks. You'll identify the various orchestration tools, but will focus on Airflow -- one of the most popular and widely used tools in the field today. You'll explore the core components of Airflow, the Airflow UI, and how to create and manage DAGs using various Airflow features.

What's included

11 videos5 readings1 assignment3 app items

Instructor

Instructor ratings
4.9 (17 ratings)
Joe Reis

Top Instructor

DeepLearning.AI
4 Courses16,960 learners

Offered by

DeepLearning.AI

Recommended if you're interested in Cloud Computing

Why people choose Coursera for their career

Felipe M.
Learner since 2018
"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."
Jennifer J.
Learner since 2020
"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."
Larry W.
Learner since 2021
"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."
Chaitanya A.
"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."

Learner reviews

4.7

47 reviews

  • 5 stars

    87.50%

  • 4 stars

    4.16%

  • 3 stars

    0%

  • 2 stars

    4.16%

  • 1 star

    4.16%

Showing 3 of 47

IH
5

Reviewed on Nov 15, 2024

TS
5

Reviewed on Nov 23, 2024

RA
5

Reviewed on Nov 19, 2024

New to Cloud Computing? Start here.

Placeholder

Open new doors with Coursera Plus

Unlimited access to 7,000+ world-class courses, hands-on projects, and job-ready certificate programs - all included in your subscription

Advance your career with an online degree

Earn a degree from world-class universities - 100% online

Join over 3,400 global companies that choose Coursera for Business

Upskill your employees to excel in the digital economy

Frequently asked questions