Data Processing with Azure

This course is part of Microsoft Azure: AI, Infrastructure, and Data Solutions Specialization

Instructor: LearnQuest Network

Access provided by Birlasoft

7,810 already enrolled

8 modules

Gain insight into a topic and learn the fundamentals.

67 reviews

Intermediate level

Some related experience required

1 week to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

8 modules

Gain insight into a topic and learn the fundamentals.

67 reviews

Intermediate level

Some related experience required

1 week to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

What you'll learn

Configure batch processing with Databricks and Data Factory on Azure
Use ETL and ELT to load and transform data
Create linked services and identify pipelines for data stored within Data Factory
Explain Data Virtualization in PolyBase

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

7 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your subject-matter expertise

This course is part of the Microsoft Azure: AI, Infrastructure, and Data Solutions Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

There are 8 modules in this course

This Azure training course is designed to equip students with the knowledge need to process, store and analyze data for making informed business decisions. Through this Azure course, the student will understand what big data is along with the importance of big data analytics, which will improve the students mathematical and programming skills. Students will learn the most effective method of using essential analytical tools such as Python, R, and Apache Spark.

This Azure training course is designed to equip the students with the knowledge need to process, store and analyze data for making informed business decisions. Through this Azure course, the student will understand what big data is along with the importance of big data analytics, which will improve the students mathematical and programming skills. Students will learn the most effective method of using essential analytical tools such as R, and Apache Spark.

What's included

1 video

One of the primary benefits of Azure Databricks is its ability to integrate with many other data environments to pull data through an ETL or ELT process. In module course, we examine each of the E, L, and T to learn how Azure Databricks can help ease us into a cloud solution.

What's included

5 videos3 readings1 assignment1 plugin

5 videos Total 40 minutes

1.1 Batch Processing with Databricks and Data Factory in Azure 8 minutes
1.2 - ELT Processing using Azure 9 minutes
1.3 - Databricks and Azure Spark 7 minutes
1.4 Transform Data using Databricks in ADF 8 minutes
1.5 Use Case: ADF and Spark 7 minutes

3 readings Total 50 minutes

Azure Databricks and Apache Spark 10 minutes
Exercise 1 - Use Batch Processing with Databricks and Data Factory on Azure 10 minutes
Exercise 2 - Intro to Databricks and Data Factory Page 30 minutes

1 assignment Total 30 minutes

Module 1 Quiz 30 minutes

1 plugin Total 15 minutes

Microsoft Video: Cloud Computing Overview 15 minutes

Processing big data in real-time is now an operational necessity for many businesses. Azure Stream Analytics is Microsoft’s serverless real-time analytics offering for complex event processing. In this section we examine how customers unlock valuable insights and gain competitive advantage by harnessing the power of big data.

What's included

4 videos3 readings1 assignment2 plugins

4 videos Total 46 minutes

Pipelines and Activities - Introduction 21 minutes
Processing using a Pipeline 7 minutes
Analyzing Logs for an HDInsight Cluster 10 minutes
Using Azure Blob Storage within HDInsight 8 minutes

3 readings Total 35 minutes

Using Azure Blob Storage with HDInsight 5 minutes
Exercise 1 - Pipeline Activities & Usage in Azure Data Factory 15 minutes
Exercise 2 - Examine Logs within the HDInsight/Blob Storage 15 minutes

1 assignment Total 30 minutes

Module 2 Quiz 30 minutes

2 plugins Total 25 minutes

Microsoft Video: Azure SQL Data Warehouse 15 minutes
2.2 Activity 10 minutes

A data factory can have one or more pipelines. A pipeline is a logical grouping of activities that together perform a task. The activities in a pipeline define actions to perform on your data. Before you create a dataset, you must create a linked service to link your data store to the data factory. This section deals with linked services and data sets within Azure Blob Storage.

What's included

5 videos1 reading1 assignment2 plugins

5 videos Total 38 minutes

Link Services and Datasets - Introduction 14 minutes
Identifying Pipelines for a Data Factory 10 minutes
Data Stores and Azure Blob Storage 5 minutes
Linked Service and Connecting Data Factory to External Resources 4 minutes
Processing Input Blobs with Azure Data Factory 6 minutes

1 reading Total 10 minutes

Exercise 1 - Link Data within Datasets in Azure Storage 10 minutes

1 assignment Total 30 minutes

Module 3 Quiz 30 minutes

2 plugins Total 25 minutes

3.2 Activity 10 minutes
Video: Processing Input Blobs with Azure Data Factory 15 minutes

Azure Data Factory is a fully managed, cloud-based data orchestration service that enables data movement and transformation. In this section, we explore scheduling triggers for Azure Data Factory to automate your pipeline execution.

What's included

5 videos1 reading1 assignment2 plugins

5 videos Total 17 minutes

Schedules and Triggers - Introduction 5 minutes
Creating a Trigger that Runs a Pipeline on a Schedule 2 minutes
Scheduling a Trigger in Azure Data Factory 4 minutes
Pipeline Execution and Triggers in ADF 3 minutes
Use Case: Azure Schedule, Trigger, and Events 3 minutes

1 reading Total 5 minutes

Pipeline Execution and Triggers in ADF 5 minutes

1 assignment Total 30 minutes

Module 4 Quiz 30 minutes

2 plugins Total 30 minutes

Video: Azure Data Factory Event Triggers 15 minutes
Module 4 Exercise 15 minutes

In time-streaming scenarios, performing operations on the data contained in temporal windows is a common pattern. Stream Analytics has native support for windowing functions, enabling developers to author complex stream processing jobs with minimal effort. In this section, we study windowing functions related to in-stream analytics.

What's included

5 videos1 reading1 assignment3 plugins

5 videos Total 30 minutes

Selecting Windowing Functions - Introduction 5 minutes
How Stream Analytics Support Native Windowing Functions 10 minutes
Temporal Windows 6 minutes
Using Window Functions in the GROUP BY Clause 3 minutes
Aggregating Events over Multiple Windows using WindowsQ 6 minutes

1 reading Total 10 minutes

Understanding Stream Analytics Windowing Functions 10 minutes

1 assignment Total 30 minutes

Module 5 Quiz 30 minutes

3 plugins Total 45 minutes

Video: Window Functions for Analysis 15 minutes
5.5 Activity 10 minutes
Module 5 Exercise 20 minutes

This section teaches how to analyze phone call data using Azure Stream Analytics. The phone call data, generated by a client application, contains some fraudulent calls, which will be filtered by the Stream Analytics job.

What's included

7 videos1 reading1 assignment3 plugins

7 videos Total 33 minutes

How Stream Analytics Relate to Data Solutions 3 minutes
Generate Sample Call Data and Send it to Event Hubs 10 minutes
Creating a Stream Analytics Job 7 minutes
Configuring Job Input and Output 4 minutes
Define a Query to Filter Fraudulent Calls 2 minutes
Test and Start the Job 4 minutes
Visualize Results in Power BI 3 minutes

1 reading Total 10 minutes

Output Real-Time Stream Analytics Data to a Power BI Dashboard 10 minutes

1 assignment Total 30 minutes

Module 6 Quiz 30 minutes

3 plugins Total 35 minutes

Video: Create a Stream Analytics Job 15 minutes
6.3 Activity 10 minutes
6.7 Activity 10 minutes

Traditional SMP data warehouses use an Extract, Transform and Load (ETL) process for loading data. Azure SQL Data Warehouse is a massively parallel processing (MPP) architecture that takes advantage of the scalability and flexibility of compute and storage resources. Utilizing an Extract, Load, and Transform (ELT) process can take advantage of MPP and eliminate resources needed to transform the data prior to loading.