Duke University
Spark, Hadoop, and Snowflake for Data Engineering
Duke University

Spark, Hadoop, and Snowflake for Data Engineering

Noah Gift
Kennedy Behrman
Matt Harrison

Instructors: Noah Gift

8,989 already enrolled

Included with Coursera Plus

Gain insight into a topic and learn the fundamentals.
4.0

(42 reviews)

Advanced level

Recommended experience

29 hours to complete
3 weeks at 9 hours a week
Flexible schedule
Learn at your own pace
Gain insight into a topic and learn the fundamentals.
4.0

(42 reviews)

Advanced level

Recommended experience

29 hours to complete
3 weeks at 9 hours a week
Flexible schedule
Learn at your own pace

What you'll learn

  • Create scalable data pipelines (Hadoop, Spark, Snowflake, Databricks) for efficient data handling.

  • Optimize data engineering with clustering and scaling to boost performance and resource use.

  • Build ML solutions (PySpark, MLFlow) on Databricks for seamless model development and deployment.

  • Implement DataOps and DevOps practices for continuous integration and deployment (CI/CD) of data-driven applications, including automating processes.

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

21 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Placeholder

Build your subject-matter expertise

This course is part of the Applied Python Data Engineering Specialization
When you enroll in this course, you'll also be enrolled in this Specialization.
  • Learn new concepts from industry experts
  • Gain a foundational understanding of a subject or tool
  • Develop job-relevant skills with hands-on projects
  • Earn a shareable career certificate
Placeholder
Placeholder

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV

Share it on social media and in your performance review

Placeholder

There are 4 modules in this course

In this module, you will learn how to work with different data engineering platforms, such as Hadoop and Spark, and apply their concepts to real-world scenarios. First, you will explore the fundamentals of Hadoop to store and process big data. Next, you will delve into Spark concepts, distributed computing, deferred execution, and Spark SQL. By the end of the week, you will gain hands-on experience with PySpark DataFrames, DataFrame methods, and deferred execution strategies.

What's included

10 videos9 readings7 assignments2 discussion prompts2 ungraded labs

In this module, you will explore the Snowflake platform, gaining insights into its architecture and key concepts. Through hands-on practice in the Snowflake Web UI, you'll learn to create tables, manage warehouses, and use the Snowflake Python Connector to interact with tables. By the end of this week, you'll solidify your understanding of Snowflake's architecture and practical applications, emerging with the ability to effectively navigate and leverage the platform for data management and analysis.

What's included

8 videos5 readings6 assignments

In this module, you will practice the essential skills for seamlessly managing machine learning workflows using Databricks and MLFlow. First, you will create a Databricks workspace and configure a cluster, setting the stage for efficient data analysis. Next, you will load a sample dataset into the Databricks workspace using the power of PySpark, enabling data manipulation and exploration. Finally, you will install MLFlow either locally or within the Databricks environment, gaining the ability to orchestrate the entire machine learning lifecycle. By the end of this week, you will be able to craft, track, and manage machine learning experiments within Databricks, ensuring precision, reproducibility, and optimal decision-making throughout your data-driven journey.

What's included

16 videos7 readings4 assignments1 ungraded lab

In this module, you will explore the concepts of Kaizen, DevOps, and DataOps and how these methodologies synergistically contribute to efficient and seamless data engineering workflows. Through practical examples, you will learn how Kaizen's continuous improvement philosophy, DevOps' collaborative practices, and DataOps' focus on data quality and integration converge to enhance the development, deployment, and management of data engineering platforms. By the end of this week, you will have the knowledge and perspective needed to optimize data engineering processes and deliver scalable, reliable, and high-quality solutions.

What's included

21 videos6 readings4 assignments1 ungraded lab

Instructors

Instructor ratings
3.8 (12 ratings)
Noah Gift
Duke University
40 Courses144,918 learners

Offered by

Duke University

Recommended if you're interested in Machine Learning

Why people choose Coursera for their career

Felipe M.
Learner since 2018
"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."
Jennifer J.
Learner since 2020
"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."
Larry W.
Learner since 2021
"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."
Chaitanya A.
"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."

Learner reviews

Showing 3 of 42

4.0

42 reviews

  • 5 stars

    52.38%

  • 4 stars

    19.04%

  • 3 stars

    11.90%

  • 2 stars

    4.76%

  • 1 star

    11.90%

VD
5

Reviewed on Jan 15, 2024

MK
5

Reviewed on Aug 6, 2024

New to Machine Learning? Start here.

Placeholder

Open new doors with Coursera Plus

Unlimited access to 7,000+ world-class courses, hands-on projects, and job-ready certificate programs - all included in your subscription

Advance your career with an online degree

Earn a degree from world-class universities - 100% online

Join over 3,400 global companies that choose Coursera for Business

Upskill your employees to excel in the digital economy

Frequently asked questions