Delve into the two different approaches to converting raw data into analytics-ready data. One approach is the Extract, Transform, Load (ETL) process. The other contrasting approach is the Extract, Load, and Transform (ELT) process. ETL processes apply to data warehouses and data marts. ELT processes apply to data lakes, where the data is transformed on demand by the requesting/calling application.
In this course, you will learn about the different tools and techniques that are used with ETL and Data pipelines. Both ETL and ELT extract data from source systems, move the data through the data pipeline, and store the data in destination systems. During this course, you will experience how ELT and ETL processing differ and identify use cases for both. You will identify methods and tools used for extracting the data, merging extracted data either logically or physically, and for loading data into data repositories. You will also define transformations to apply to source data to make the data credible, contextual, and accessible to data users. You will be able to outline some of the multiple methods for loading data into the destination system, verifying data quality, monitoring load failures, and the use of recovery mechanisms in case of failure. By the end of this course, you will also know how to use Apache Airflow to build data pipelines as well be knowledgeable about the advantages of using this approach. You will also learn how to use Apache Kafka to build streaming pipelines as well as the core components of Kafka which include: brokers, topics, partitions, replications, producers, and consumers. Finally, you will complete a shareable final project that enables you to demonstrate the skills you acquired in each module.
YC
Love the labs, but do not like the robotic lectures.
SK
Relevant information in recordings, good recap of every video and hand-on lesson in the end to concrete the knowledge.
BN
Overall it's a good course. I wish I could use dos2unix, tr, or sed for removing ^M from the toll_data.tsv. The Final Assignment Instructions could have been clearer.
OG
It takes 1 hour to connect the lab and start the service.
HT
Course offers valuable conceptual content but labs could be improved. Coursera assessment system is really poor.
TK
Great hands-on (alike the others in the pack)! Practical and interactive.
AW
Great Course, Assignments prepare really well and flexible
RS
Succinctly presented. Labs really hammered the point home :)
MD
Great learning course for Kafka/ Airflow, well presented
DR
Good introduction to Airflow and Kafka however only one airflow operator is explored
MT
Great content but the labs can be challenging to work with.
MA
Thanks to all the instructor's efforts, one of the best DATA engineering courses, contains hands-on Experience with essential data tools.
Showing: 20 of 98
Course content is good but labs are riddled with bugs and in dire need of quality control. I encountered many time-consuming, frustrating technical issues that made completing this course a slog. Final assignment introduces some difficult linux manipulations that were not covered in the coures and are not really that relevant to the subject matter. Some questions on the final are unclear and could be better written. Would recommend the instructors or whomever created this course to eat their own cooking and go through this course and fix the various issues.
Buggy practice. Not possible to complete without fixing airflow start script yourself. Nobody monitor or fixing issues here
I cannot proceed with the "SUBMIT a DAG" lab as I am constantly being shown the error - "cp: cannot create regular file '/home/project/airflow/dags/my_first_dag.py': Permission denied" when I run the command - "cp my_first_dag.py $AIRFLOW_HOME/dags".
How are you expecting me to complee this lab when I am getting a permission denied error. Please fix this asap.
The course is really basic, it only introduces the keywords and very high level concepts of ETL. Barely discusses any technical challenges or constraints. Some of the questions in the quizzes are absurd.
Thanks to IBM and Coursera for the great "ETL and Data Pipelines with Shell, Airflow and Kafka" course, that I passed with Grade Achieved: 100%. It's the third course, that I've passed, as a part of "IBM Data Engineering Specialization". I was so carried away by the course that I literally sat up until 2 am almost every day. In this course I could apply my knowledge of Python, Pandas, SQL, Bash commands to build ETL Batch and Stream pipelines.
The final project to connect Airflow as a pipeline management tool to Kafka server is a very useful hands-on project. More details or explanations on the syntax of Python calling Kafka producer and consumer, which are in the files of toll_traffic_generator.py and streaming_data_reader.py, would be more valuable rather than just providing these two files to run on its own.
It's one of the most challenging courses I've been enrolled!
REALLY A GOOD COURSE BUT:
- Labs are not debugged (inaccuracies)
- I understand that Kafka a wide technology and maybe it will take more than a week to cover in a appropiate way, but the labs were only about copy and paste commands.
Love the labs, but do not like the robotic lectures.
I can't understand some of the questions in quizzes. Moreover, the peer-to-peer grading system should be converted to automatic grading.
Perfect environment to make experiments! Very easy and powerful in use.
As introductory level course this is well packed course. Off course it would have been better if the audio is not AI generated - the pace of speaking is not optimal, and can't really be adjusted to optimal. Maybe there is too much emphasis on bash scripting. Theoretical concepts, especially for Kafka can be better explained, especially scenarios with multiple topics, groups, brokers, failure scenarios. And in general reading materials are big point for improvement. The lab environment could have been more stable and updated with newer packages, but it is nothing to cry over.
It's great introduction for airflow and kafka but still an introduction it is shallow doesn't offer much but at the end you will understand what you need to continue further in both technologies.
final quiz sucks!
why are you so sick to make up this quiz ?
this quiz is pretty much more like GMAT reading test instead of IT assessment !
This is a well-crafted course about ETL and building/streaming Data Pipelines. The hands-on labs experience including practicing shell scripting prepared well for the final assignment task. Writing the real-world-scenario DAG tasks to create the ETL Data Pipelines using Apache Airflow was challenging. Successfully submitting the DAG and monitoring it in the UI DAGs list was at the same time satisfactory.
Excellent introduction to this topics. Labs contain all you need to know how to start using this type of technologies. Highly recommended.
Interesting course with enough labs.
i really liked the labs
A decent overview of Airflow and Kafka. Worth it for the time invested. The labs were good, however the execution of the final assignment was poor -- you have to submit two dozen screen captures for a peer reviewed assignment. Taking screen caps of code is silly, why not just submit the code? Plus you are taking the caps before you even know if your code works. And you are relying on strangers to read and understand your code before you can get credit for the course. Fortunately, some kind soul found mine quickly and gave me 100%. My code did work -- I tested it thoroughly -- but you can't really tell from screen caps.
A good introductory course to airflow and kafka. Could have been broken up into at least two courses focusing on each of these platform, and going more in depth in each one. Also, the final assignment is a pain to complete especially due to some errors in instructions. But overall, It is a decent course.