Coursera

Building Smarter Data Pipelines: SQL, Spark, Kafka & GenAI Specialization

Ends in 5 days! Save 40% on your access to 10,000+ programs and make a real impact in your career. Save now.

Coursera

Building Smarter Data Pipelines: SQL, Spark, Kafka & GenAI Specialization

Build Scalable Data Engineering Systems.

Learn to design, implement, and optimize data pipelines using industry-standard tools and frameworks

Caio Avelino
Starweaver
Soheil Haddadi

Instructors: Caio Avelino +6 more

2,202 already enrolled

Included with Coursera Plus

Get in-depth knowledge of a subject

from 101 reviews of courses in this program

Intermediate level

Recommended experience

4 weeks to complete
at 10 hours a week
Flexible schedule
Learn at your own pace
Get in-depth knowledge of a subject

from 101 reviews of courses in this program

Intermediate level

Recommended experience

4 weeks to complete
at 10 hours a week
Flexible schedule
Learn at your own pace

What you'll learn

  • Design and implement scalable data ingestion, processing, and storage systems using Apache Kafka and Spark

  • Build high-performance data pipelines integrating cloud platforms, databases, and generative AI technologies

  • Apply data engineering best practices for enterprise-scale analytics, optimization, and real-time processing

Skills you'll gain

  • Category: Data Governance
  • Category: SQL
  • Category: Cloud Computing Architecture
  • Category: Real Time Data
  • Category: Business Intelligence
  • Category: Data Pipelines
  • Category: Responsible AI
  • Category: Extract, Transform, Load
  • Category: Star Schema
  • Category: Data Quality
  • Category: Performance Tuning
  • Category: Data Modeling
  • Category: Data Warehousing
  • Category: Quality Management
  • Category: Snowflake Schema

Tools you'll learn

  • Category: Microsoft SQL Servers
  • Category: Apache Kafka
  • Category: Apache Hadoop
  • Category: Generative AI
  • Category: Apache Spark

Details to know

Shareable certificate

Add to your LinkedIn profile

Taught in English

Advance your subject-matter expertise

  • Learn in-demand skills from university and industry experts
  • Master a subject or tool with hands-on projects
  • Develop a deep understanding of key concepts
  • Earn a career certificate from Coursera

Specialization - 9 course series

Engineering Data Ecosystems: Pipelines, ETL, Spark

Engineering Data Ecosystems: Pipelines, ETL, Spark

Course 1, 3 hours

What you'll learn

  • Identify and describe the components and importance of data ecosystems.

  • Understand the basic structure and function of data pipelines.

  • Recognize the steps involved in ETL workflows and their role in data handling.

  • Gain an introductory knowledge of big data and the application of Apache Spark.

Skills you'll gain

Category: Apache Spark
Category: Extract, Transform, Load
Category: Data Processing
Category: Data Integration
Category: Dataflow
Category: Big Data
Category: Data Pipelines
Category: Data Management
Data Engineering: Pipelines, ETL, Hadoop

Data Engineering: Pipelines, ETL, Hadoop

Course 2, 3 hours

What you'll learn

  • Analyse the architecture and components of data pipelines to understand their impact on data flow and processing efficiency.

  • Implement robust ETL processes, for scalability and maintainability.

  • Analyze big data challenges and introduce Hadoop ecosystem tools (HDFS, MapReduce, Hive, Pig, and Spark) for data processing tasks.

Skills you'll gain

Category: Data Pipelines
Category: Extract, Transform, Load
Category: Apache Hadoop
Category: Data Architecture
Category: Data Analysis
Category: Data Transformation
Category: Big Data
Category: Dataflow
Category: Data Strategy
Category: Data-Driven Decision-Making
Category: Apache Hive
Category: Data Management
Category: Apache Spark
Category: Scalability
Category: Data Warehousing
Category: Data Integration
Category: Data Processing
Data Warehousing: Schema, ETL, Optimal Performance

Data Warehousing: Schema, ETL, Optimal Performance

Course 3, 3 hours

What you'll learn

  • Explain the importance of data warehousing in business intelligence.

  • Design and implement effective schema designs for data warehouses.

  • Implement ETL processes to load and transform data into a data warehouse.

  • Apply performance optimization techniques to enhance data warehouse efficiency.

Skills you'll gain

Category: Data Warehousing
Category: Query Languages
Category: Snowflake Schema
Category: Extract, Transform, Load
Category: Star Schema
Category: Databases
Category: Performance Tuning
Category: Data Transformation
Category: Data Management
Category: Data Modeling
Category: Performance Improvement
Category: Data Integration
Category: Database Design
Category: Business Intelligence
Microsoft SQL Server: Performance Tuning Essentials

Microsoft SQL Server: Performance Tuning Essentials

Course 4, 3 hours

What you'll learn

  • Analyze and tune SQL queries to enhance SQL performance and reduce application latency.

  • Evaluate effective database index and maintenance task strategies to improve efficiency.

  • Monitor the performance of troubleshooting techniques used for resolving common SQL server issues.

  • Apply best practices for SQL Server performance to ensure consistent and reliable operations.

Skills you'll gain

Category: SQL
Category: Microsoft SQL Servers
Category: Performance Tuning
Category: Query Languages
Category: Database Management
Category: System Monitoring
Category: Application Performance Management
Category: Database Management Systems
Apache Kafka - An Introduction

Apache Kafka - An Introduction

Course 5, 3 hours

What you'll learn

  • Describe Apache Kafka's architecture and its components, enhancing data pipeline efficiency.

  • Configure and manage Kafka clusters, ensuring high availability and fault tolerance.

  • Apply (Create and use) topics, publishers, and subscribers to facilitate real-time data exchange.

  • Implement basic stream processing applications using Kafka Streams, addressing real-world data challenges.

Skills you'll gain

Category: Real Time Data
Category: Scalability
Category: Apache Kafka
Category: Data Processing
Category: Data Pipelines
Cloud Architecture Design Patterns

Cloud Architecture Design Patterns

Course 6, 3 hours

What you'll learn

  • Show understanding of the fundamentals of cloud architecture, including key components like virtual machines, storage, and networking.

  • Identify and implement core cloud design patterns such as Load Balancer, Circuit Breaker, and Auto-Scaling to ensure scalability and reliability.

  • Demonstrate advanced cloud design patterns, including Microservices Architecture, Event-Driven Architecture, and Serverless Computing.

Skills you'll gain

Category: Serverless Computing
Category: Load Balancing
Category: Cloud Computing Architecture
Category: Microservices
Category: Cloud Infrastructure
Category: Infrastructure As A Service (IaaS)
Category: Software Architecture
Category: Cloud Computing
Category: Software Design Patterns
Category: Cloud Services
Category: Scalability
Category: Event-Driven Programming
Category: Cloud Deployment
Design Real-Time Architectures with Spark & Kafka

Design Real-Time Architectures with Spark & Kafka

Course 7, 4 hours

What you'll learn

  • Examine core real-time data principles and how Kafka and Spark support streaming architectures.

  • Create real-time pipelines by connecting Kafka topics with Spark Structured Streaming.

  • Improve and deploy streaming systems using monitoring, fault tolerance, and tuning.

Skills you'll gain

Category: Apache Kafka
Category: Real Time Data
Category: Apache Spark
Category: Software Architecture
Category: Performance Tuning
Category: Systems Architecture
Category: Data Processing
Category: Real-Time Operating Systems
Category: Application Deployment
Category: Distributed Computing
Category: Data Pipelines
Category: Architecture and Construction
Category: Performance Management
Category: Event-Driven Programming
Category: Scalability
Category: Data Transformation
Smart Data Cleaning with Generative AI

Smart Data Cleaning with Generative AI

Course 8, 7 hours

What you'll learn

Skills you'll gain

Category: Generative AI
Category: Data Cleansing
Category: Automation
Category: OpenAI
Category: Tensorflow
Category: Data Preprocessing
Category: Alteryx
Category: Responsible AI
Category: Data Validation
Category: Data Quality
GenAI for Data Engineers: Scaling with GenAI

GenAI for Data Engineers: Scaling with GenAI

Course 9, 3 hours

What you'll learn

  • Identify the capabilities of GenAI for basic role specific, Data Engineer functions.

  • Examine real-world applications to leverage GenAI for streamlining work and fostering innovation in Data Engineering functions.

  • Deploy strategies and tactics to responsibly integrate GenAI into data engineering practices, while maintaining human oversight and accountability.

Skills you'll gain

Category: Generative AI
Category: Data Modeling
Category: Responsible AI
Category: Data Pipelines
Category: Data Transformation
Category: SQL

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV. Share it on social media and in your performance review.

Instructors

Caio Avelino
Caio Avelino
9 Courses8,992 learners
Starweaver
Starweaver
Coursera
573 Courses1,166,983 learners
Soheil Haddadi
Soheil Haddadi
Coursera
6 Courses6,056 learners

Offered by

Coursera

Why people choose Coursera for their career

Felipe M.

Learner since 2018
"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."

Jennifer J.

Learner since 2020
"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."

Larry W.

Learner since 2021
"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."

Chaitanya A.

"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."

Frequently asked questions