Managing Big Data in Clusters and Cloud Storage

This course is part of the Modern Big Data Analysis with SQL Specialization

4.7
stars
185 ratings

Ian Cook +1 more instructor
Top Instructor

Offered By

Modern Big Data Analysis with SQL SpecializationCloudera

About this Course

In this course, you'll learn how to manage big datasets, how to load them into clusters and cloud storage, and how to apply structure to the data so that you can run queries on it using distributed SQL engines like Apache Hive and Apache Impala. You’ll learn how to choose the right data types, storage systems, and file formats based on which tools you’ll use and what performance you need.
By the end of the course, you will be able to
• use different tools to browse existing databases and tables in big data systems;
• use different tools to explore files in distributed big data filesystems and cloud storage;
• create and manage big data databases and tables using Apache Hive and Apache Impala; and
• describe and choose among different data types and file formats for big data systems.

To use the hands-on environment for this course, you need to download and install a virtual machine and the software on which to run it. Before continuing, be sure that you have access to a computer that meets the following hardware and software requirements:
• Windows, macOS, or Linux operating system (iPads and Android tablets will not work)
• 64-bit operating system (32-bit operating systems will not work)
• 8 GB RAM or more
• 25GB free disk space or more
• Intel VT-x or AMD-V virtualization support enabled (on Mac computers with Intel processors, this is always enabled;
on Windows and Linux computers, you might need to enable it in the BIOS)
• For Windows XP computers only: You must have an unzip utility such as 7-Zip or WinZip installed (Windows XP’s built-in unzip utility will not work)

Shareable Certificate

Earn a Certificate upon completion

100% online

Start instantly and learn at your own schedule.

Course 3 of 3 in the

Modern Big Data Analysis with SQL Specialization

Flexible deadlines

Reset deadlines in accordance to your schedule.

Beginner Level

Approx. 21 hours to complete

English

Subtitles: Arabic, French, Portuguese (European), Italian, Vietnamese, German, Russian, English, Spanish

What you will learn

Use different tools to browse existing databases and tables in big data systems

Use different tools to explore files in distributed big data filesystems and cloud storage

Create and manage big data databases and tables using Apache Hive and Apache Impala

Describe and choose among different data types and file formats for big data systems

Skills you will gain

Data ManagementDistributed File SystemsCloud StorageBig DataSQL

Shareable Certificate

Earn a Certificate upon completion

100% online

Start instantly and learn at your own schedule.

Course 3 of 3 in the

Modern Big Data Analysis with SQL Specialization

Flexible deadlines

Reset deadlines in accordance to your schedule.

Beginner Level

Approx. 21 hours to complete

English

Subtitles: Arabic, French, Portuguese (European), Italian, Vietnamese, German, Russian, English, Spanish

Instructors

Instructor rating

4.88/5 (28 Ratings)

Ian Cook
Top Instructor

Staff Curriculum Developer

Cloudera

17,784 Learners

2 Courses

Glynn Durham

Offered by

Cloudera

At Cloudera, we believe that data can make what is impossible today, possible tomorrow. We empower people to transform complex data into clear and actionable insights. Cloudera delivers an enterprise data cloud for any data, anywhere, from the Edge to AI. Powered by the relentless innovation of the open source community, Cloudera advances digital transformation for the world’s largest enterprises.

Syllabus - What you will learn from this course

Week

Week 1

3 hours to complete

Orientation to Data in Clusters and Cloud Storage

3 hours to complete

7 videos (Total 56 min), 3 readings, 1 quiz

7 videos

Welcome to the Course4m

Browsing Tables with Hue7m

Browsing Tables with SQL Utility Statements6m

Browsing HDFS with the Hue File Browser13m

Browsing HDFS from the Command Line9m

Understanding S3 and Other Cloud Storage Platforms6m

Browsing S3 Buckets from the Command Line8m

3 readings

Review and Preparation30m

Instructions for Downloading and Installing the Exercise Environment30m

Troubleshooting the VM5m

1 practice exercise

Week 1 Graded Quiz30m

Week

Week 2

5 hours to complete

Defining Databases, Tables, and Columns

5 hours to complete

7 videos (Total 33 min), 12 readings, 2 quizzes

7 videos

Week 2 Introduction3m

Introduction to the CREATE TABLE Statement5m

Using Different Schemas on the Same Data12m

Specifying TBLPROPERTIES2m

Examining, Modifying, and Removing Tables1m

Hive and Impala Interoperability2m

Impala Metadata Refresh3m

12 readings

Creating Databases and Tables with Hue30m

Creating Databases and Tables with SQL15m

Permissions to Create Databases and Tables5m

The ROW FORMAT Clause25m

The STORED AS Clause15m

The LOCATION Clause20m

CREATE TABLE Shortcuts10m

Using Hive SerDes15m

Working with Unstructured and Semi-Structured Data15m

Examining Table Structure10m

Dropping Databases and Tables5m

Modifying Existing Tables35m

2 practice exercises

Week 2 Practice Quiz30m

Week 2 Graded Quiz30m

Week

Week 3

3 hours to complete

Data Types and File Types

3 hours to complete

5 videos (Total 14 min), 12 readings, 2 quizzes

5 videos

Week 3 Introduction1m

Overview of Data Types1m

Choosing the Right Data Types4m

Overview of File Types3m

Choosing the Right File Types3m

12 readings

Integer Data Types5m

Decimal Data Types10m

Character String Data Types10m

Other Data Types5m

Examining Data Types10m

Out-of-Range Values5m

Text Files5m

Avro Files5m

Parquet Files5m

ORC Files5m

Other File Types5m

Creating Tables with Avro and Parquet Files20m

2 practice exercises

Week 3 Practice Quiz20m

Week 3 Graded Quiz30m

Week

Week 4

5 hours to complete

Managing Datasets in Clusters and Cloud Storage

5 hours to complete

8 videos (Total 48 min), 13 readings, 3 quizzes

8 videos

Week 4 Introduction3m

Refresh Impala's Metadata Cache after Loading Data2m

Loading Files into HDFS with Hue's Table Browser10m

Loading Files into HDFS with Hue's File Browser6m

Loading Files into HDFS from the Command Line8m

Loading Files into S3 from the Command Line10m

Using Hive and Impala to Load Data into Tables3m

Conclusion2m

13 readings

More about HDFS Shell Commands10m

Chaining and Scripting with HDFS Commands5m

HDFS Permissions5m

Other Ways to Load Files into S35m

S3 Permissions10m

Missing Values15m

Character Sets5m

Using Sqoop to Import Data15m

More Sqoop Import Options5m

Using Sqoop to Export Data5m

SQL LOAD DATA Statements10m

SQL INSERT Statements10m

SQL INSERT ... SELECT and CTAS Statements15m

2 practice exercises

Week 4 Practice Quiz30m

Week 4 Graded Quiz30m

Reviews

4.7

42 reviews

5 stars
78.37%
4 stars
16.21%
3 stars
4.32%
2 stars
0.54%
1 star
0.54%

TOP REVIEWS FROM MANAGING BIG DATA IN CLUSTERS AND CLOUD STORAGE

by PZSep 3, 2020

This was definitely the most challenging course I have done so far, but I am loving it! Glynn and Ian do a fantastic job of guiding you through the content in an engaging and practical manner.

by SYFeb 29, 2020

This is Very good course for a beginners, it gives you lots of exercises to practice in vm and course material is Really really good but only thing is you have to read a lot ,

by AMDec 15, 2019

It would have been nice if videos would have been present instead of reading. Also a more deep diving would have been done in concepts like bucketing and indexing.

by MAApr 13, 2020

This is one of the systematic specializations which makes the harder and otherwise overwhelming subject so easy to navigate, follow and learn.

View all reviews

About the Modern Big Data Analysis with SQL Specialization

This Specialization teaches the essential skills for working with large-scale data using SQL.

Maybe you are new to SQL and you want to learn the basics. Or maybe you already have some experience using SQL to query smaller-scale data with relational databases. Either way, if you are interested in gaining the skills necessary to query big data with modern distributed SQL engines, this Specialization is for you. Most courses that teach SQL focus on traditional relational databases, but today, more and more of the data that’s being generated is too big to be stored there, and it’s growing too quickly to be efficiently stored in commercial data warehouses. Instead, it’s increasingly stored in distributed clusters and cloud storage. These data stores are cost-efficient and infinitely scalable. To query these huge datasets in clusters and cloud storage, you need a newer breed of SQL engine: distributed query engines, like Hive, Impala, Presto, and Drill. These are open source SQL engines capable of querying enormous datasets. This Specialization focuses on Hive and Impala, the most widely deployed of these query engines. This Specialization is designed to provide excellent preparation for the Cloudera Certified Associate (CCA) Data Analyst certification exam. You can earn this certification credential by taking a hands-on practical exam using the same SQL engines that this Specialization teaches—Hive and Impala.

Frequently Asked Questions

When will I have access to the lectures and assignments?
Access to lectures and assignments depends on your type of enrollment. If you take a course in audit mode, you will be able to see most course materials for free. To access graded assignments and to earn a Certificate, you will need to purchase the Certificate experience, during or after your audit. If you don't see the audit option:
The course may not offer an audit option. You can try a Free Trial instead, or apply for Financial Aid.
The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.
What will I get if I subscribe to this Specialization?
When you enroll in the course, you get access to all of the courses in the Specialization, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile. If you only want to read and view the course content, you can audit the course for free.
Is financial aid available?
Yes, Coursera provides financial aid to learners who cannot afford the fee. Apply for it by clicking on the Financial Aid link beneath the "Enroll" button on the left. You'll be prompted to complete an application and will be notified if you are approved. You'll need to complete this step for each course in the Specialization, including the Capstone Project. Learn more.
What are the hardware and software requirements for the exercise environment?
• Windows, macOS, or Linux operating system (iPads and Android tablets will not work) • 64-bit operating system (32-bit operating systems will not work) • 8 GB RAM or more • 25GB free disk space or more • Intel VT-x or AMD-V virtualization support enabled (on Mac computers with Intel processors, this is always enabled; on Windows and Linux computers, you might need to enable it in the BIOS) • For Windows XP computers only: You must have an unzip utility such as 7-Zip or WinZip installed (Windows XP’s built-in unzip utility will not work)

More questions? Visit the Learner Help Center.

Managing Big Data in Clusters and Cloud Storage