Introduction to Data Engineering

Introduction to Data Engineering

This course is part of multiple programs.

Instructors: Rav Ahuja +1 more

265,433 already enrolled

Included with

4 modules

Gain insight into a topic and learn the fundamentals.

3,642 reviews

Beginner level

Recommended experience

Flexible schedule

1 week at 10 hours a week

Learn at your own pace

95%

Most learners liked this course

4 modules

Gain insight into a topic and learn the fundamentals.

3,642 reviews

Beginner level

Recommended experience

Flexible schedule

1 week at 10 hours a week

Learn at your own pace

95%

Most learners liked this course

What you'll learn

List basic skills required for an entry-level data engineering role.
Discuss various stages and concepts in the data engineering lifecycle.
Describe data engineering technologies such as Relational Databases, NoSQL Data Stores, and Big Data Engines.
Summarize concepts in data security, governance, and compliance.

Skills you'll gain

Category: Data Science
Category: Data Architecture
Category: Data Governance
Category: Data Pipelines
Category: Data Store
Category: Data Infrastructure
Category: Databases
Category: Data Integration
Category: Big Data
Category: SQL
Category: Extract, Transform, Load
Category: Data Storage Technologies
Category: Database Design
Category: Data Warehousing
Category: Data Storage

Tools you'll learn

Category: Apache Spark
Category: NoSQL
Category: Relational Databases
Category: Data Lakes
Category: Apache Hadoop

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

23 assignments¹

AI Graded see disclaimer

Taught in English

Build your subject-matter expertise

This course is available as part of

When you enroll in this course, you'll also be asked to select a specific program.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

There are 4 modules in this course

Start your journey in one of the fastest growing professions today with this beginner-friendly Data Engineering course! You will be introduced to the core concepts, processes, and tools you need to know in order to get a foundational knowledge of data engineering. as well as the roles that Data Engineers, Data Scientists, and Data Analysts play in the ecosystem.

You will begin this course by understanding what is data engineering as well as the roles that Data Engineers, Data Scientists, and Data Analysts play in this exciting field. Next you will learn about the data engineering ecosystem, the different types of data structures, file formats, sources of data, and the languages data professionals use in their day-to-day tasks. You will become familiar with the components of a data platform and gain an understanding of several different types of data repositories such as Relational (RDBMS) and NoSQL databases, Data Warehouses, Data Marts, Data Lakes and Data Lakehouses. You’ll then learn about Big Data processing tools like Apache Hadoop and Spark. You will also become familiar with ETL, ELT, Data Pipelines and Data Integration. This course provides you with an understanding of a typical Data Engineering lifecycle which includes architecting data platforms, designing data stores, and gathering, importing, wrangling, querying, and analyzing data. You will also learn about security, governance, and compliance. You will learn about career opportunities in the field of Data Engineering and the different paths that you can take for getting skilled as a Data Engineer. You will hear from several experienced Data Engineers, sharing their insights and advice. By the end of this course, you will also have completed several hands-on labs and worked with a relational database, loaded data into the database, and performed some basic querying operations.

In this module, you will learn about the different entities that come together to form a modern data ecosystem and the role Data Engineers, Data Scientists, Data Analysts, Business Analysts, and Business Intelligence Analysts play in this ecosystem. You will learn what data engineering is and the key tasks in a data engineering lifecycle. You will also gain an understanding of the responsibilities of a data engineer, the skillsets they need in order to be successful, and what a typical day in the life of a data engineer looks like.

What's included

10 videos2 readings6 assignments

10 videosTotal 56 minutes

Welcome to Introduction to Data Engineering3 minutes
Modern Data Ecosystem 5 minutes
Key Players in the Data Ecosystem6 minutes
Specializations in Data Engineering 10 minutes
What is Data Engineering?4 minutes
Viewpoints: Defining Data Engineering 4 minutes
Viewpoints: Evolution of Data Engineering 8 minutes
Responsibilities and Skillsets of a Data Engineer5 minutes
Viewpoints: Skills and Qualities to be a Data Engineer7 minutes
A Day in the Life of a Data Engineer4 minutes

2 readingsTotal 20 minutes

Summary and Highlights10 minutes
Summary and Highlights10 minutes

6 assignmentsTotal 72 minutes

Graded Quiz: Data Engineering20 minutes
Graded Quiz: Application of Data Engineering Skills16 minutes
Practice Quiz: Data Engineering Fundamentals8 minutes
Practice Quiz: Understanding Data Roles and Responsibilities10 minutes
Practice Quiz: Data Engineering Evolution and Roles8 minutes
Practice Quiz: Data Engineering Responsibilities and Skills10 minutes

In this module, you will learn about the data engineering ecosystem, the different types of data structures, file formats, sources of data, and the languages data professionals use in their day-to-day tasks. You will gain an understanding of several different types of data repositories such as relational and non-relational databases, data warehouses, data marts, and data lakes. You will learn about ETL and ELT processes, data pipelines, and data integration platforms. You will also gain an understanding of what big data is, and the tools used for processing and storing big data. At the end of this module, you will be guided to create an IBM Cloud account, and provision an instance of IBM Db2.

What's included

18 videos7 readings6 assignments1 app item

18 videosTotal 113 minutes

Overview of the Data Engineering Ecosystem5 minutes
Types of Data4 minutes
Understanding Different Types of File Formats 5 minutes
Sources of Data8 minutes
Languages for Data Professionals9 minutes
Viewpoints: Working with Varied Data Sources and Types7 minutes
Overview of Data Repositories5 minutes
RDBMS8 minutes
NoSQL8 minutes
Data Warehouses, Data Marts, and Data Lakes7 minutes
(Optional): Data Lakehouses Explained9 minutes
Viewpoints: Considerations for Choice of Data Repository6 minutes
ETL, ELT, and Data Pipelines7 minutes
Data Integration Platforms5 minutes
Viewpoints: Tools, Databases, and Data Repositories of Choice7 minutes
Foundations of Big Data5 minutes
Big Data Processing Tools: Hadoop, HDFS, Hive, and Spark7 minutes
Viewpoints: Impact of Big Data on Data Engineering4 minutes

7 readingsTotal 70 minutes

Metadata and Metadata Management10 minutes
Summary and Highlights10 minutes
Summary and Highlights10 minutes
Optional Labs for IBM Cloud and Db210 minutes
[Optional] Hands-on Lab: Create your IBM Cloud account10 minutes
[Optional ] Hands-on Lab: Provision an instance of IBM Db2 Lite plan10 minutes
Summary and Highlights10 minutes

6 assignmentsTotal 57 minutes

Graded Quiz10 minutes
Graded Quiz10 minutes
Graded Quiz10 minutes
Practice Quiz10 minutes
Practice Quiz12 minutes
Practice Quiz5 minutes

1 app itemTotal 60 minutes

[Optional] Obtain IBM Cloud Feature Code and Activate Trial Account60 minutes

In this module, we will walk you through the data engineering lifecycle. You will learn about the architecture of a data platform, factors for selecting and designing data stores, and the different facets of security as it applies to data platforms and data lifecycle management. You will also learn about the process, steps, and tools used for gathering, importing, wrangling, and querying data. You will gain an understanding of performance monitoring and the steps you can take to troubleshoot performance issues. We will also talk about governance regulations, why we need them, and how technology enables compliance to regulations. During the course of this module, you will be guided to load data from a CSV file into the IBM Db2 instance you created in the previous module. You will also be guided to explore your dataset using some basic SQL queries that will be provided to you.

What's included

10 videos7 readings8 assignments2 app items

10 videosTotal 64 minutes

Architecting the Data Platform7 minutes
Factors for Selecting and Designing Data Stores7 minutes
Security6 minutes
Viewpoints: Importance of Data Security4 minutes
How to Gather and Import Data7 minutes
Data Wrangling7 minutes
Tools for Data Wrangling6 minutes
Querying and Analyzing Data6 minutes
Performance Tuning and Troubleshooting7 minutes
Governance and Compliance8 minutes

7 readingsTotal 80 minutes

Summary and Highlights10 minutes
[Optional] Hands-on Lab: Load data into the Db2 Database from a CSV file15 minutes
Summary and Highlights10 minutes
[Optional] Hands-on Lab: Explore Your Dataset Using SQL Queries in DB215 minutes
Summary and Highlights10 minutes
Summary and Highlights10 minutes
Optional: Overview of the DataOps Methodology10 minutes

8 assignmentsTotal 54 minutes

Graded Quiz10 minutes
Graded Quiz10 minutes
Graded Quiz 10 minutes
Graded Quiz5 minutes
Practice Quiz6 minutes
Practice Quiz6 minutes
Practice Quiz5 minutes
Practice Quiz2 minutes

2 app itemsTotal 30 minutes

Hands-On Lab: Load data into the Datasette from a CSV file15 minutes
Lab: Explore your dataset using SQL queries using Datasette15 minutes

In this module, you will learn about career opportunities in the field of Data Engineering and the different paths that you can take for getting skilled as a Data Engineer. At the end of the module, you will be presented with the final graded assignment which is divided into two parts. The first part consists of a set of quiz questions, and the second part includes open-ended questions that will be reviewed and graded through an AI-graded, auto-scored assignment.

What's included

6 videos5 readings3 assignments1 app item

6 videosTotal 36 minutes

Career Opportunities in Data Engineering6 minutes
Viewpoints: Get into Data Engineering8 minutes
Data Engineering Learning Path4 minutes
Viewpoints: What Do Employers Look for in a Data Engineer6 minutes
Viewpoints: The Many Paths to Data Engineering6 minutes
Viewpoints: Advice to Aspiring Data Engineers7 minutes

5 readingsTotal 39 minutes

Data Warehousing Specialist15 minutes
Data Manager10 minutes
Summary and Highlights10 minutes
Final Project Submission Guidelines and Deliverables2 minutes
Congratulations and Next Steps2 minutes

3 assignmentsTotal 75 minutes

Graded Quiz20 minutes
Final Quiz50 minutes
Practice Quiz5 minutes

1 app itemTotal 20 minutes

AI-Graded - Final Submission and Evaluation20 minutes

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV. Share it on social media and in your performance review.

Instructors

Instructor ratings

(1,219 ratings)

Rav Ahuja

IBM

55 Courses5,024,913 learners

Offered by

IBM

Why people choose Coursera for their career

Felipe M.

Learner since 2018

"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."

Jennifer J.

Learner since 2020

"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."

Larry W.

Learner since 2021

"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."

Chaitanya A.

"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."

Learner reviews

5 stars
79.88%
4 stars
16.38%
3 stars
2.30%
2 stars
0.43%
1 star
0.98%

Showing 3 of 3642

Reviewed on Apr 7, 2024

Very good introduction to Data Engineering. Not too technical, but a very good foundation of both technical concepts and soft skills required in becoming a successful data engineer.

Reviewed on Sep 25, 2021

I felt the course was the perfect introduction for someone like me that has been on the sales side of most software. I feel that it wasnt ovewhelming at all, and I mostly enjoyed the labs.

Reviewed on Dec 2, 2022

This course was well made and informative. I went in knowing little about data engineering and now I have a much clear understanding of what data engineering is all about.

Frequently asked questions

To access the course materials, assignments and to earn a Certificate, you will need to purchase the Certificate experience when you enroll in a course. You can try a Free Trial instead, or apply for Financial Aid. The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.

When you enroll in the course, you get access to all of the courses in the Certificate, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile.

¹ Some assignments in this course are AI-graded. For these assignments, your data will be used in accordance with Coursera's Privacy Notice.