Move storage off-cluster with Google Cloud Storage - Recommending Products using Cloud SQL and Spark | Coursera

Explore all of Coursera

Browse
Top Courses
Log In
Join for Free

Move storage off-cluster with Google Cloud Storage

Loading...

Google Cloud Platform Big Data and Machine Learning Fundamentals

4.6 (10,509 ratings) | 120K Students Enrolled

Course 1 of 6 in the Data Engineering with Google Cloud Specialization

Enroll for Free

This 2-week accelerated on-demand course introduces participants to the Big Data and Machine Learning capabilities of Google Cloud Platform (GCP). It provides a quick overview of the Google Cloud Platform and a deeper dive of the data processing capabilities. At the end of this course, participants will be able to: • Identify the purpose and value of the key Big Data and Machine Learning products in the Google Cloud Platform • Use CloudSQL and Cloud Dataproc to migrate existing MySQL and Hadoop/Pig/Spark/Hive workloads to Google Cloud Platform • Employ BigQuery and Cloud Datalab to carry out interactive data analysis • Choose between Cloud SQL, BigTable and Datastore • Train and use a neural network using TensorFlow • Choose between different data processing products on the Google Cloud Platform Before enrolling in this course, participants should have roughly one (1) year of experience with one or more of the following: • A common query language such as SQL • Extract, transform, load activities • Data modeling • Machine learning and/or statistics • Programming in Python Google Account Notes: • Google services are currently unavailable in China.

Skills You'll Learn

Tensorflow, Bigquery, Google Cloud Platform, Cloud Computing

Reviews

4.6 (10,509 ratings)

5 stars
71%
4 stars
24%
3 stars
4%
2 stars
1%
1 star
1%

RT

May 07, 2020

It is a fun journey to explore the Big Data and ML resource in GCP. And even cooler is you don't need work for a big company or own lots of resource to get your hands dirty to support your learning!

BV

Dec 24, 2019

It was very good training with some of the real-time use cases enjoyed a lot. As a new person to google cloud and big data, I think this the best basic fundamental which I have come across so far.

From the lesson

Recommending Products using Cloud SQL and Spark

In this module you will have an existing Apache SparkML recommendation model that is running on-premise. You will learn about recommendation models and how you can run them in the cloud with Cloud Dataproc and Cloud SQL.

Approach: Move from on-premise to Google Cloud Platform9:50

Demo: From zero to an Apache Spark job in 10 minutes or less6:20

Challenge: Utilizing and tuning on-premise clusters6:05

Move storage off-cluster with Google Cloud Storage4:54

Taught By

Google Cloud Training

Try the Course for Free

Transcript

As you might remember from the demo earlier, separating compute and storage enables cost-effective scale for workloads. With the notion of turning down clusters, you may be worried about all the data that's currently stored on disk HDFS. You may be using HDFS on-premises. What happens to that with our new Google Cloud Platform architecture? Your data is not stored in the cluster, your data is now stored off-cluster in Google Cloud Storage buckets. Really, won't that make things slow to reach out across a network, each time the cluster needs some data? Recall that earlier we mentioned Google's data center bandwidth between compute and storage. We talked about a petabit bisectional bandwidth. What that means is that, if you draw a line somewhere in the network the bisectional bandwidth is the rate of communication at which servers on one side of the line can communicate with servers on the other side. With enough bisectional bandwidth, any server can communicate with any other server at full network speeds. With petabit bisectional bandwidth, the communication is so fast that it no longer makes sense to transfer the files and store them locally. Instead, it makes sense to use the data from where it is stored. So here's the plan. We'll use HDFS, but we'll use HDFS only on the cluster for working storage, storage during processing, but we will store all actual input and output data on Google Cloud storage. Because the input and output data is off the cluster, the cluster can be created for a single job or type of workload and can be shut down when not in use. Changing your code that works on prem to have it work with the data on cloud storage is easy. Just replace HDFS in your Spark or Big Cloud with GS. So you take the HDFS URLs, HDFS :// and replace it with GS :// This will make the Spark of big job read or write to cloud storage, that's it. There's also an H-Base connector for Cloud Bigtable. So if you have your data in Bigtable you can use the HBase connector. There's a Big Query connector that you can use to work with data, if the data is in the analytics warehouse. So when we say off-cluster storage, we're talking about cloud storage or we are talking about BigQuery or we are talking about Bigtable. Cloud Data-proc and Google cloud storage are tightly coupled with other GCP products. So storing your data off cluster, means that you can process this data not just with Spark from Data-proc but also from all these other products. Not to mention, storing it off cluster in GCS is generally cheaper, since: a, disks attached to computer instances are expensive in and of themselves and b, if your data are off cluster, you'd get to shutdown the computer nodes when you're not using them. So bottom line, store your data in Google Cloud Storage. Friends don't let friends use HDFS. So let's recap what we've covered. You can get a cloud data-proc cluster up and running in about 90 seconds. This gives you all of the power of Hadoop without having to manage clusters. As you saw, you can lift-and-shift your existing Hadoop and Spark workloads, by simply replacing HDFS:// URLs with GS:// URLs. You can connect Cloud Data-proc to Google Cloud Storage and unlock the benefits of both scale and cloud economics. You get to provision a cluster per job if you want to, and shut the cluster down when you're done. Lastly, your clusters are customizable, which could include autoscaling and preemptible VMs so that you get cost savings.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Top Online Courses

AI for Everyone
Introduction to TensorFlow
Neural Networks and Deep Learning
Algorithms, Part 1
Algorithms, Part 2
Machine Learning
Machine Learning with Python
Machine Learning Using Sas Viya
R Programming
Intro to Programming with Matlab
Data Analysis with Python
AWS Fundamentals: Going Cloud Native
Google Cloud Platform Fundamentals
Site Reliability Engineering
Speak English Professionally
The Science of Well Being
Learning How to Learn
Financial Markets
Hypothesis Testing in Public Health
Foundations of Everyday Leadership

Top Online Specializations

Deep Learning
Python for Everybody
Data Science
Applied Data Science with Python
Business Foundations
Architecting with Google Cloud Platform
Data Engineering on Google Cloud Platform
Excel to MySQL
Advanced Machine Learning
Mathematics for Machine Learning
Self-Driving Cars
Blockchain Revolution for the Enterprise
Business Analytics
Excel Skills for Business
Digital Marketing
Statistical Analysis with R for Public Health
Fundamentals of Immunology
Anatomy
Managing Innovation and Design Thinking
Foundations of Positive Psychology

Online Certificates

Google IT Support
IBM Customer Engagement Specialist
IBM Data Science
Applied Project Management
IBM Applied AI Professional Certificate
Machine Learning for Analytics
Spatial Data Analysis and Visualization
Construction Engineering and Management
Instructional Design

Online Degree Programs

Master's in Data Science
Bachelors Degree in Computer Science
Computer Science and Engineering Degrees
Master's in Machine Learning
MBA and Business Degrees
Master's in Electrical Engineering
Master's in Public Health
Master's in Information Technology

Coursera

О проекте
Руководство
Карьера
Каталог
Сертификаты
Сертификаты MasterTrack™
Степени
Для организаций
Для правительственных организаций
Для стационара
Борьба с последствиями коронавируса

Community

Учащиеся
Партнеры
Разработчики
Бета-тестировщики
Переводчики
Блог
Технический блог

More

Условия
Конфиденциальность
Помощь
Доступность
Пресса
Контакты
Справочник
Филиалы

Get it on Google Play

© 2020 Coursera Inc. All rights reserved.