Monitoring, Streams, and Connectors

Monitoring, Streams, and Connectors

This course is part of Apache Kafka Specialization

Instructor: LearnKartS

Access provided by Abu Dhabi National Oil Company

5 modules

Gain insight into a topic and learn the fundamentals.

Intermediate level

Recommended experience

4 hours to complete

Flexible schedule

Learn at your own pace

5 modules

Gain insight into a topic and learn the fundamentals.

Intermediate level

Recommended experience

4 hours to complete

Flexible schedule

Learn at your own pace

What you'll learn

Monitor brokers, producers, and consumer lag in your Kafka setup
Build real-time apps with Kafka Streams using tasks and stateful processing
Connect Kafka to other tools with REST-based connectors
Use Schema Registry and monitor clusters for smooth operations

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

5 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your subject-matter expertise

This course is part of the Apache Kafka Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

There are 5 modules in this course

Want to be sure your Kafka pipelines are running at top performance?

This Kafka Monitoring, Streams, and Connectors course teaches you how to design, monitor, and manage real-time data systems with precision. The course is designed for learners with Kafka and Java/Scala expertise. It covers broker, producer, and consumer metrics, latency monitoring, quota enforcement, and alarm configuration using JMX and Prometheus. You'll learn about Kafka Streams' architecture, including stateful and stateless transformations, K-Stream/K-Table joins, windowing, and exactly-once semantics. You will also learn how to deploy Kafka Connect, configure source/sink connectors, interact with single message transformations (SMTs), and manage connectors using the Kafka Connect REST API. This Apache Kafka course teaches stream monitoring and integration from start to finish with demos, videos, readings, and evaluations. By the end of this course, you'll be able to create and monitor high-throughput Kafka pipelines. Enroll now to gain real-time data expertise.

This module introduces Kafka monitoring concepts and explains why observability is critical in distributed systems. It covers broker, topic, and consumer metrics, along with lag tracking and dashboards for production insights.

What's included

10 videos1 reading1 assignment

10 videosTotal 43 minutes

Monitoring and Its Importance2 minutes
Server or Infrastructure Monitoring and Application Monitor2 minutes
Kafka Monitoring 3 minutes
Kafka Broker Metrics - Under Replicated Partitions6 minutes
Kafka Broker Metrics - Others9 minutes
Topic and Partition Specific Metrics2 minutes
Logging and Client Monitoring3 minutes
Producer and Consumer Metrics7 minutes
Quotas and Lag Monitoring6 minutes
Monitoring Dashboard4 minutes

1 readingTotal 10 minutes

Flashcards: Kafka Metrics and What They Indicate10 minutes

1 assignmentTotal 12 minutes

Kafka Monitoring and Metrics12 minutes

This module focuses on schema management and data consistency using Kafka Schema Registry. It also covers schema evolution, compatibility rules, and real-world integration patterns.

What's included

6 videos1 reading1 assignment

6 videosTotal 28 minutes

Kafka Schema Registry6 minutes
Kafka Component and Architecture4 minutes
Kafka Schema Registry - Internal working and Use Cases4 minutes
Kafka Schema Registry Working5 minutes
DEMO: Setting up an Open-Source Health Monitor6 minutes
DEMO: Using Kafka Schema Registry with Kafka4 minutes

1 readingTotal 10 minutes

Cheat Sheet: Schema Registry API and Compatibility Modes10 minutes

1 assignmentTotal 12 minutes

Kafka Schema Registry12 minutes

This module introduces Kafka Streams architecture and core streaming concepts. It covers stream processing model, state stores, threading, and how real-time data pipelines work.

What's included

6 videos1 assignment

This module explores Kafka Streams DSL and APIs for building stream applications. It includes KStream, KTable, transformations, aggregations, windowing, and processor topology.

What's included

8 videos1 reading1 assignment

8 videosTotal 45 minutes

Kafka Stream DSL8 minutes
K-Stream Operations8 minutes
Introduction to K-Table5 minutes
Understanding K-Table Operations 4 minutes
Aggregation and Windowing6 minutes
Processor Topology and Stream Processor3 minutes
Stream and Processor APIs8 minutes
Processor APIs - Create Topology4 minutes

1 readingTotal 10 minutes

RESOURCE: Kafka Streams DSL API Reference10 minutes

1 assignmentTotal 12 minutes

Kafka Streams12 minutes

This module explains Kafka Connect architecture and its role in system integration. It covers standalone and distributed modes, REST API management, and building source and sink connectors.