Learn>
Data Science>
Machine Learning>
Google Cloud Platform Big Data and Machine Learning Fundamentals course>
Move storage off-cluster with Google Cloud Storage

Loading...

Move storage off-cluster with Google Cloud Storage

In this module you will have an existing Apache SparkML recommendation model that is running on-premise. You will learn about recommendation models and how you can run them in the cloud with Cloud Dataproc and Cloud SQL.

Google Cloud Platform Big Data and Machine Learning Fundamentals

Course 1 of 5 in the Data Engineering on Google Cloud Platform Specialization

This 1-week accelerated on-demand course introduces participants to the Big Data and Machine Learning capabilities of Google Cloud Platform (GCP). It provides a quick overview of the Google Cloud Platform and a deeper dive of the data processing capabilities. At the end of this course, participants will be able to: • Identify the purpose and value of the key Big Data and Machine Learning products in the Google Cloud Platform • Use CloudSQL and Cloud Dataproc to migrate existing MySQL and Hadoop/Pig/Spark/Hive workloads to Google Cloud Platform • Employ BigQuery and Cloud Datalab to carry out interactive data analysis • Choose between Cloud SQL, BigTable and Datastore • Train and use a neural network using TensorFlow • Choose between different data processing products on the Google Cloud Platform Before enrolling in this course, participants should have roughly one (1) year of experience with one or more of the following: • A common query language such as SQL • Extract, transform, load activities • Data modeling • Machine learning and/or statistics • Programming in Python Google Account Notes: • Google services are currently unavailable in China.

Skills You'll Learn

Tensorflow, Bigquery, Google Cloud Platform, Cloud Computing

Taught By

Google Cloud Training

Transcript

As you might remember from the demo earlier,

separating computer storage enables

cost-effective scale for workloads.

With the notion of turning down clusters,

you may be worried about all the data that's

currently stored on disk HDFS.

You may be using HDFS on-premises.

What happens to that with

a new Google Cloud Platform architecture?

Your data is not stored in the cluster,

your data is now stored

off-cluster in Google Cloud Storage buckets.

Really, would make things

slow to reach out across a network,

each time the cluster needs some data?

Recall that earlier we mentioned

Google's data center bandwidth between computer storage.

We talked about a petabyte bi-sectional bandwidth.

What that means is that,

if you draw a line somewhere in

the network the bi-sectional bandwidth is the rate of

communication at which servers on one side

of the line can

communicate with servers on the other side.

With enough bi-sectional bandwidth,

any server can communicate with

any other server at full network speeds.

With petabyte bi-sectional bandwidth,

the communication is so fast that it no longer

makes sense to transfer the files and store them locally.

Instead, it makes sense to

use the data from where it is stored.

So here's the plan, we'll use HDFS.

But we'll use only on

the cluster for working

storage,storage during processing,

but we will store

all actual input and output data on Google Cloud storage.

Because the input and output data is off the cluster,

the cluster can be created for a single job or

type of workload and can be shut down when not in use.

Changing your code that works on prem to

have it work with the data on cloud storage is easy.

Just replace HDFS in your spark a big cloud with GS.

So you take the HDFS URLs,

HDFS :// and replace it the GS

:// This will make the spark of

big job read or write to cloud storage, that's it.

There's also an H-Base connector for Cloud Bigtable.

So if you have your data in

Bigtable you can use the HBase connector.

There's a big query connector

that you can use to work with data,

if the data is in the analytics warehouse.

So when we say off-cluster storage,

we're talking about cloud storage or we are talking

about big query or we are talking about bigtable.

Cloud Data-proc and Google cloud storage are

tightly coupled with other GCP products.

So storing your data off cluster,

means that you can process this data not just with spark

Data-proc but also from all these other products.

Not to mention, storing it off cluster in

GCS is generally cheaper, since; a,

Disks attached to computer instances are

expensive in an [inaudible] and b,

if your data are off cluster,

you'd get to shutdown the computer nodes

when you're not using them.

So bottom line, store your data in Google Cloud Storage.

Friends don't let friends use HDFS.

So let's recap what we've covered.

You can get a cloud data-proc cluster

up and running in about 90 seconds.

This gives you all of the power of Hadoop

without having to manage clusters.

As you saw, you can lift and shift

your existing Hadoop and spark workloads,

by simply replacing HDFS:// URLs with GS:// URLs.

You can connect Cloud Data-proc

to Google Cloud Storage and

unlock the benefits of both scale and cloud economics.

You get to provision a cluster per job if you want to,

and shut the cluster down when you're done.

Lastly, your clusters are customizable,

which could include auto-scaling and pre-emptible VMs

so that you get cost savings.

About Coursera

Courses, Specializations, and Online Degrees taught by top instructors from the world's best universities and educational institutions.

Join a community of 40 million learners from around the world

Earn a skill-based course certificate to apply your knowledge

Gain confidence in your skills and further your career

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2019 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play