Data Engineering Capstone Project

Data Engineering Capstone Project

This course is part of IBM Data Engineering Professional Certificate

Instructors: Rav Ahuja

Access provided by Coursera Learning Team

19,518 already enrolled

7 modules

Gain insight into a topic and learn the fundamentals.

142 reviews

Advanced level

Recommended experience

2 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

7 modules

Gain insight into a topic and learn the fundamentals.

142 reviews

Advanced level

Recommended experience

2 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

What you'll learn

Demonstrate proficiency in skills required for an entry-level data engineering role.
Design and implement various concepts and components in the data engineering lifecycle such as data repositories.
Showcase working knowledge with relational databases, NoSQL data stores, big data engines, data warehouses, and data pipelines.
Apply skills in Linux shell scripting, SQL, and Python programming languages to Data Engineering problems.

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

14 assignments¹

AI Graded see disclaimer

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your Data Management expertise

This course is part of the IBM Data Engineering Professional Certificate

When you enroll in this course, you'll also be enrolled in this Professional Certificate.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate from IBM

There are 7 modules in this course

Showcase your skills in this Data Engineering project! In this course you will apply a variety of data engineering skills and techniques you have learned as part of the previous courses in the IBM Data Engineering Professional Certificate.

You will demonstrate your knowledge of Data Engineering by assuming the role of a Junior Data Engineer who has recently joined an organization and be presented with a real-world use case that requires architecting and implementing a data analytics platform. In this Capstone project you will complete numerous hands-on labs. You will create and query data repositories using relational and NoSQL databases such as MySQL and MongoDB. You’ll also design and populate a data warehouse using PostgreSQL and IBM Db2 and write queries to perform Cube and Rollup operations. You will generate reports from the data in the data warehouse and build a dashboard using Cognos Analytics. You will also show your proficiency in Extract, Transform, and Load (ETL) processes by creating data pipelines for moving data from different repositories. You will perform big data analytics using Apache Spark to make predictions with the help of a machine learning model. This course is the final course in the IBM Data Engineering Professional Certificate. It is recommended that you complete all the previous courses in this Professional Certificate before starting this course.

In this module, you will design a data platform that uses MySQL as an OLTP database. You will be using MySQL to store the OLTP data.

What's included

1 video2 assignments1 app item4 plugins

1 video Total 4 minutes

Introduction to Capstone Project 4 minutes

2 assignments Total 36 minutes

Checklist: OLTP Database 24 minutes
Graded Quiz: OLTP Database 12 minutes

1 app item Total 30 minutes

Lab: OLTP Database 30 minutes

4 plugins Total 45 minutes

Reading: Final Project Submission Guidelines and Deliverables 15 minutes
Data Platform Architecture 10 minutes
Assignment Overview: OLTP Database 15 minutes
OLTP Database Requirements and Design 5 minutes

In this module, you will design a data platform that uses MongoDB as a NoSQL database. You will use MongoDB to store the e-commerce catalog data.

What's included

2 assignments1 app item1 plugin

In this module you will design and implement a data warehouse and you will then generate reports from the data in the data warehouse.

What's included

3 assignments2 app items1 plugin

3 assignments Total 69 minutes

Checklist: Data Warehouse Design & Setup 15 minutes
Checklist: Data Warehouse Reporting 24 minutes
Graded Quiz: Build a Data Warehouse 30 minutes

2 app items Total 120 minutes

Hands-on Lab: Data Warehousing 60 minutes
Hands-on Lab: Data Warehouse Reporting using PostgreSQL 60 minutes

1 plugin Total 15 minutes

Assignment Overview:Data Warehouse Design and Reporting 15 minutes

In this module, you will assume the role of a data engineer at an e-commerce company. Your company has finished setting up a data warehouse. Now you are assigned the responsibility to design a reporting dashboard that reflects the key metrics of the business.

What's included

5 readings2 assignments6 plugins

5 readings Total 42 minutes

(Optional): About this optional lesson with Looker Studio 2 minutes
(Optional) : Getting Started with Google Looker Studio 10 minutes
(Optional): Creating Visualizations in Reports using Looker Studio 10 minutes
(Optional) : Summary and Highlights 10 minutes
Final Assignment Overview 10 minutes

2 assignments Total 27 minutes

Checklist: Dashboard Creation 12 minutes
Graded Quiz: Dashboard Creation 15 minutes

6 plugins Total 210 minutes

Assignment Overview: Data Analytics 15 minutes
(Optional):Hands-on Lab: Getting Started with Google Looker Studio 60 minutes
(Optional): Hands-on Lab: Creating and Configuring Visualizations in Reports with Google Looker Studio 60 minutes
(Optional) Hands-on Lab: Advanced charts in Looker Studio 15 minutes
(Optional): Final Assignment : Dashboard Creation using IBM Cognos Analytics 30 minutes
(Optional): Final Assignment : Dashboard Creation using Google Looker Studio 30 minutes

In this module, you will perform ETL operations to move transactional data from an OLTP database (MySQL) into a data warehouse (PostgreSQL).Finally, you will implement and automate an ETL pipeline in Python that extracts daily incremental records from the production database, transforms them as needed, and loads them into the warehouse. Once the ETL process is established, you will extend it further using Apache Airflow, a powerful workflow orchestration tool. You will design DAGs (Directed Acyclic Graphs) that define task dependencies, automate the extraction and transformation of web server logs, and archive processed data for downstream analytics.

What's included

3 assignments2 app items1 plugin

In this module, you will use the data from a webserver to analyse search terms. You will then load a pretrained sales forecasting model and predict the sales forecast for a future year.

What's included

2 assignments2 app items1 plugin

In this module, you will make a final submission of all the labs you’ve completed throughout the course for evaluation.You can choose to have your submission evaluated by an AI tool or through a peer-graded review.