Data Engineering with Scala and Spark

Data Engineering with Scala and Spark

Instructor: Packt - Course Instructors

Access provided by SR University

13 modules

Gain insight into a topic and learn the fundamentals.

Intermediate level

Recommended experience

2 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

13 modules

Gain insight into a topic and learn the fundamentals.

Intermediate level

Recommended experience

2 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

What you'll learn

Set up a development environment for building data pipelines in Scala
Use Spark DataFrames, Datasets, and SQL with Scala for data processing
Profile and clean data using Deequ for improved data quality

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

13 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

There are 13 modules in this course

This course is designed to equip data engineers with the skills to build scalable and efficient data pipelines using Scala and Spark. Data engineers will learn best practices for development, testing, and deployment in cloud environments, with a focus on optimizing performance and ensuring data quality. The course provides the necessary tools to transform raw data into actionable insights, making it highly relevant in today’s data-driven world.

Throughout the course, learners will improve their data engineering skills by mastering techniques for building both streaming and batch data pipelines. The content emphasizes practical outcomes such as performance tuning and data profiling. With hands-on examples and step-by-step guidance, learners will gain a solid understanding of real-time and batch processing pipelines. What makes this course unique is its combination of foundational theory and real-world applications. By the end, you will be able to use Scala and Spark to process large datasets and optimize pipelines in cloud environments effectively. This course is ideal for data engineers with some experience in data processing. While it assumes familiarity with data engineering concepts and cloud technologies, anyone eager to improve their skills in Scala and Spark will benefit from the practical, step-by-step approach.

In this section, we explore functional programming, higher-order functions, polymorphic functions, and pattern matching in Scala for data engineering applications.

What's included

2 videos6 readings1 assignment

2 videos Total 2 minutes

Course Overview 1 minute
Scala Essentials for Data Engineers - Overview Video 1 minute

6 readings Total 120 minutes

Introduction 10 minutes
Understanding Objects, Classes, and Traits 10 minutes
Trait 10 minutes
Examples of HOFs from the Scala Collection Library 30 minutes
Understanding Polymorphic Functions 30 minutes
Understanding Pattern Matching 30 minutes

1 assignment Total 10 minutes

Scala Essentials for Data Engineers 10 minutes

In this section, we explore cloud-based and local environments for data engineering pipelines, focusing on setup processes, trade-offs, and practical applications.

What's included

1 video5 readings1 assignment

In this section, we explore Apache Spark's APIs, focusing on DataFrame and Dataset for distributed data processing.

What's included

1 video3 readings1 assignment

In this section, we explore using Spark JDBC API for database access, designing database interfaces, and performing operations with configuration loading.

What's included

1 video3 readings1 assignment

In this section, we explore object stores, data lakes, and lakehouses, focusing on their roles in managing large-scale data workflows efficiently.

What's included

1 video6 readings1 assignment

In this section, we explore Spark transformations, aggregations, joins, and window functions to enhance data processing for BI and analytics. Key concepts include efficient data manipulation and pipeline development.

What's included

1 video4 readings1 assignment

In this section, we explore Deequ for implementing data quality checks, analyzing completeness and accuracy, and defining constraints to ensure reliable data pipelines.

What's included

1 video3 readings1 assignment

In this section, we explore test-driven development, static code analysis, and linting to improve code quality, maintainability, and consistency in data engineering projects.

What's included

1 video4 readings1 assignment

1 video Total 1 minute

Test-Driven Development, Code Health, and Maintainability - Overview Video 1 minute

4 readings Total 70 minutes

Introduction 20 minutes
Performing Integration Testing 10 minutes
Running Static Code Analysis 30 minutes
Understanding Linting and Code Style 10 minutes

1 assignment Total 10 minutes

Test-Driven Development and Code Maintainability Fundamentals 10 minutes

In this section, we explore CI/CD practices with GitHub to automate Scala data pipeline workflows, focusing on GitHub Actions, version control, and reliable deployment processes.

What's included

1 video4 readings1 assignment

In this section, we explore data pipeline orchestration using tools like Airflow, Argo, Databricks, and Azure Data Factory. We focus on workflow design, task management, and real-world implementation strategies.

What's included

1 video6 readings1 assignment

1 video Total 1 minute

Data Pipeline Orchestration - Overview Video 1 minute

6 readings Total 80 minutes

Introduction 10 minutes
Monitoring and UI 10 minutes
Working with Argo Workflows 20 minutes
Creating an Argo Workflow 10 minutes
Using Databricks Workflows 20 minutes
Leveraging Azure Data Factory 10 minutes

1 assignment Total 10 minutes

Data Pipeline Orchestration Fundamentals 10 minutes

In this section, we analyze Spark UI metrics to identify performance issues, optimize data shuffling, and right-size compute resources for efficient data processing.

What's included

1 video4 readings1 assignment

In this section, we explore building batch pipelines using Spark and Scala, focusing on medallion architecture, data ingestion, transformation, and orchestration for scalable data processing.

What's included

1 video5 readings1 assignment

1 video Total 1 minute

Building Batch Pipelines Using Spark and Scala - Overview Video 1 minute

5 readings Total 60 minutes

Introduction 10 minutes
Understanding the Data 20 minutes
The End-to-End Pipeline 10 minutes
Transforming the Data 10 minutes
Creating a Serving Layer 10 minutes

1 assignment Total 10 minutes

Data Pipeline Fundamentals with Spark and Scala 10 minutes

In this section, we explore building real-time data pipelines using Spark, Scala, and Kafka for IoT applications. Key concepts include data ingestion, transformation, and serving layer design.