Want to make sense of the volumes of data you have collected? Need to incorporate data-driven decisions into your process? This course provides an overview of machine learning techniques to explore, analyze, and leverage data. You will be introduced to tools and algorithms you can use to create machine learning models that learn from data, and to scale those models up to big data problems. At the end of the course, you will be able to: • Design an approach to leverage data using the steps in the machine learning process. • Apply machine learning techniques to explore and prepare data for modeling. • Identify the type of machine learning problem in order to apply the appropriate set of techniques. • Construct models that learn from data using widely available open source tools. • Analyze big data problems using scalable machine learning algorithms on Spark.
Summary of Big Data Integration and Processing
Loading...
From the course by University of California San Diego
Machine Learning With Big Data
830 ratings
From the lesson
Welcome
Meet the Instructors
Mai NguyenLead for Data Analytics
San Diego Supercomputer Center
Ilkay AltintasChief Data Science Officer
San Diego Supercomputer Center
In this video, we will provide you with a quick summary of the main points
from our first three courses to recall what you have learned.
If you have just completed our third course and do not need a refresher,
you might skip to the next lecture.
We started our first course explaining how a new torrent of big data
combined with cloud computing capabilities to process data anytime and
anywhere has been at the core of the launch of the big data era.
Such capabilities enable or present opportunities for
many dynamic data-driven applications, including energy management,
smart cities, precision medicine, and smart manufacturing.
These applications are increasingly more data-driven, dynamic and
heterogeneous in terms of their technology needs.
They're also more process-driven and
need to be tackled using a collaborative approach by a team that
puts value on accountability and reproducibility of the results.
Overall, by modeling, managing, integrating diverse
data streams we add value to our big data and
improve our business even more before we start analyzing it.
A part of modeling and
managing big data is focusing on the dimensions of the scalability and
considering the challenges associated with these dimensions to pick the right tools.
We also talked about characteristics of big data, referring to some Vs
like volume, variety, velocity, veracity and valence.
Each week presents a challenging dimension of big data,
namely size, complexity, speed, quality and connectedness.
We also added a sixth V,
value, referring to the real reason we are interested in big data.
To turn it into an advantage in the context of a problem using data science
techniques, big data needs to be analyzed.
We explained a five steps process for data science that includes data acquisition,
modeling, management, integration, and analysis.
The influence of big data pushes for
alternative scalability approaches at each step of the process.
If we just focus on the scalability challenges related to the three Vs,
we can say big data has varying volume and
velocity, requiring dynamic and scalable batch and stream processing.
Big data has variety, requiring management of data in many different data systems,
and integration of it at scale.
In our introduction to the big data course,
we talked about the version of a layer diagram for the tools in the Hadoop
ecosystem, organized vertically based on the interface.
Low level interfaces for storage and scheduling on the bottom
and high level languages and interactivity at the top.
Most of the tools in the Hadoop ecosystem were initially built to compliment
the capabilities of Hadoop for distributed file system management using HDFS.
Data processing using the MapReduce engine, and
resource scheduling, and negotiation using the YARN engine.
Over time, a number of new projects were built,
either to add to these complementary tools or to handle additional types of
big data management and processing not available in Hadoop, just like Spark.
Arguably, the most important change to Hadoop over time
was the separation of YARN from the MapReduce programming model
to solely handle resource management concerns.
This allowed for Hadoop to be extensible to different programming models and enable
the development of a number of processing engines for batch and stream processing.
Another way to look at the vast number of tools that have been added to the Hadoop
ecosystem is from the point of view of their functionality
in the big data processing pipeline.
Simply put, these are associated with three distinct layers for
data management and storage, for data processing and
for resource coordination and workflow management.
In our second course, we talked in detail about the bottom layer in this diagram,
namely data management and storage.
While this layer includes Hadoop's HDFS, there are a number of other systems
that rely on HDFS as a file system or implement their own no-SQL storage option.
As big data can have a variety of structured, semi-structured, and
unstructured formats and gets analyzed through a variety of tools,
many tools were introduced to fit this variety of needs.
We call these big data management systems.
We reviewed Redis and Aerospike as
key value stores where each data item is identified with a unique key.
We also got some practical experience with Lucene and
Gephi as vector and graph-stores respectively.
We also talked about Vertica as a column-store database where
information is stored in columns rather than rows.
Cassandra and HBase are also in this category.
Finally, we introduced Solr and Asterisk DB for managing unstructured and
semi-structured text and MongoDB as a document store.
The processing layer is where all these different types
of data get retrieved, integrated, and analyzed,
which was the primary focus of our third course.
In the integration and processing layer,
we roughly refer to the tools that are built on top of HTFS and YARN,
although some of them were with other storage and file systems.
YARN is a significant enable of many of these tools making a number of batch and
stream processing engines like Storm, Spark, Flink and Beam possible.
This layer also includes tools like Hive and Spark SQL for
bringing a query interface on top of the storage layer, Pig for
scripting simple big data pipelines using the MapReduce framework and
a number of specialized analytical libraries,
formation learning, and graph analytics.
Giraph and GraphX of Spark are examples of such libraries for graph processing.
Mahout on top of the Hadoop stack and
MLlib of Spark Are two options for machine learning.
Although we had a basic overview of graph processing and
machine learning for big data analytics earlier in our second and
third courses, we haven't gone into the details there.
In this course, we will use Spark's MLlib as one of our two main tools,
providing a deeper introduction to the machine learning library of Spark.
The third and top layer in our diagram is the coordination and management layer.
This is where integrations, scheduling, coordination, and
monitoring of applications across many tools in the bottom two layers take place.
This layer is also where the results of the big data analysis get communicated
to other programs, websites, visualization tools and business intelligence tools.
Workflow management systems help to develop automated solutions that
can manage and coordinate the process of combining data management and
analytical tasks in a big data pipeline as a configurable, structured set of steps.
Workflow driven thinking also matches
this basic process of data science that we overviewed before.
Oozie is an example workflow scheduler that can interact with
many of the tools in the integration and processing layer.
Zookeeper is the resource coordination tool which monitors and
manages and coordinates all these tools and named after animal.
Now that we've reviewed all three layers we are ready to come back to
the integration and processing layer, but now in the context of machine learning.
In which we will use machine learning techniques to
apply to our five step data science process and analyze big data.
Just a simple Google search for big data processing pipelines will bring
a vast number of pipelines with a large number of technologies
that support scalable data cleaning, preparation, and analysis.
How do we make sense of all of it to make sure we use the right tools for
our application?
How do we pick the right pre-processing and
machine learning techniques to start doing predictive modeling?
Over the next few weeks Dr. Mei will walk you through some of the most
fundamental machine learning techniques, along with introductory hands
on exercises we designed for you to ease you into the world of machine learning.
Let's get started.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
