Building Modern Data Applications Using Databricks Lakehouse

Building Modern Data Applications Using Databricks Lakehouse

Instructor: Packt - Course Instructors

Access provided by NMIMS Indore

10 modules

Gain insight into a topic and learn the fundamentals.

Beginner level

Recommended experience

1 week to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

10 modules

Gain insight into a topic and learn the fundamentals.

Beginner level

Recommended experience

1 week to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

What you'll learn

Deploy near-real-time data pipelines using Delta Live Tables
Orchestrate data pipelines with Databricks workflows
Implement data validation and monitor data quality

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

10 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

There are 10 modules in this course

In today’s data-driven world, building scalable and efficient data applications is crucial for staying ahead in business and technology. This course explores the power of Databricks Lakehouse, a unified platform for managing and analyzing large volumes of data, and guides you through essential skills to create modern data applications.

Throughout the course, you’ll learn to work with Delta Live Tables for data transformation, management, and quality assurance. You will also dive deep into Databricks’ Unity Catalog for enhanced governance, data lineage, and location management. The hands-on experience with deploying and maintaining DLT pipelines using Terraform prepares you for real-world data infrastructure challenges. This course stands out by combining theoretical understanding with practical, real-world applications. You’ll gain a robust set of skills in data pipeline management, governance, and monitoring, preparing you for building production-level data applications with Databricks Lakehouse. Designed for professionals looking to deepen their expertise in modern data architecture, this course is suitable for data engineers, data scientists, and IT professionals who want to leverage Databricks to solve real-world data problems.

In this section, we explore real-time data pipelines with Delta Live Tables (DLT), analyze Delta Lake architecture, and design scalable streaming solutions for lakehouse environments

What's included

2 videos6 readings1 assignment

2 videos Total 2 minutes

Course Overview 1 minute
An Introduction to Delta Live Tables - Overview Video 1 minute

6 readings Total 60 minutes

Introduction 10 minutes
The Databricks Lakehouse 10 minutes
Views 10 minutes
Unity Catalog 10 minutes
Supporting Concurrent Table Reads and Writes 10 minutes
Tracking Table Changes Using Change Data Feed 10 minutes

1 assignment Total 10 minutes

Delta Live Tables Fundamentals 10 minutes

In this section, we cover ingesting data with DLT, applying changes, and configuring pipelines for scalability.

What's included

1 video6 readings1 assignment

1 video Total 1 minute

Applying Data Transformations Using Delta Live Tables - Overview Video 1 minute

6 readings Total 70 minutes

Introduction 10 minutes
Using Auto Loader with DLT 10 minutes
Publishing Datasets to Unity Catalog 10 minutes
Data Pipeline Settings 10 minutes
Pipeline Cluster Types 10 minutes
Loading External Dependencies 20 minutes

1 assignment Total 10 minutes

Mastering Delta Live Tables Pipelines 10 minutes

In this section, we explore implementing data quality expectations in DLT pipelines, validating data integrity with temporary datasets, and quarantining poor-quality data for correction.

What's included

1 video4 readings1 assignment

1 video Total 1 minute

Managing Data Quality Using Delta Live Tables - Overview Video 1 minute

4 readings Total 50 minutes

Introduction 10 minutes
Using Temporary Datasets to Validate Data Processing 10 minutes
Creating a New DLT Pipeline Definition 10 minutes
Acting on Failed Expectations 20 minutes

1 assignment Total 10 minutes

Data Quality Management in Delta Live Tables 10 minutes

In this section, we cover scaling DLT pipelines through cluster optimization, autoscaling, and Delta Lake techniques.

What's included

1 video4 readings1 assignment

In this section, we explore implementing data governance in a lakehouse using Unity Catalog, focusing on access controls, data discovery, and lineage tracking for compliance and security.

What's included

1 video7 readings1 assignment

1 video Total 1 minute

Mastering Data Governance in the Lakehouse with Unity Catalog - Overview Video 1 minute

7 readings Total 55 minutes

Introduction 5 minutes
A Problem Worth Solving 10 minutes
Unity Catalog-Enabled Cluster Types 5 minutes
Enabling Unity Catalog on an Existing Databricks Workspace 5 minutes
Data Discovery and Cataloging 10 minutes
Observability with System Tables 10 minutes
Hands-on Example Data Masking Healthcare Datasets 10 minutes

1 assignment Total 10 minutes

Data Governance in the Lakehouse with Unity Catalog 10 minutes

In this section, we cover managing data storage locations in Unity Catalog with secure governance and access control.

What's included

1 video3 readings1 assignment

In this section, we explore data lineage in Unity Catalog, tracing origins, visualizing transformations, and identifying dependencies to ensure data integrity and proactive issue detection.

What's included

1 video3 readings1 assignment

In this section, we cover deploying and managing DLT pipelines using Terraform in Databricks.

What's included

1 video4 readings1 assignment

In this section, we explore Databricks Asset Bundles (DABs) for streamlining data pipeline deployment, emphasizing GitHub integration, version control, and cross-team collaboration.

What's included

1 video4 readings1 assignment

1 video Total 1 minute

Leveraging Databricks Asset Bundles to Streamline Data Pipeline Deployment - Overview Video 1 minute

4 readings Total 60 minutes

Introduction 20 minutes
Databricks Asset Bundles in Action 20 minutes
Hands-on Exercise Deploying Your First DAB 10 minutes
Hands-on Exercise Simplifying Cross-Team Collaboration with GitHub Actions 10 minutes

1 assignment Total 10 minutes

Streamlining Data Pipeline Deployment with Databricks 10 minutes

In this section, we explore monitoring data pipelines using Databricks, focusing on health, performance, and data quality. Techniques include DBSQL alerts and webhook triggers for real-time issue resolution.