Offered By

Big Data Analysis with Scala and Spark (Scala 2 version)

École Polytechnique Fédérale de Lausanne

About this Course

3,022 recent views

Manipulating big data distributed over a cluster using functional concepts is rampant in industry, and is arguably one of the first widespread industrial uses of functional ideas. This is evidenced by the popularity of MapReduce and Hadoop, and most recently Apache Spark, a fast, in-memory distributed collections framework written in Scala. In this course, we'll see how the data parallel paradigm can be extended to the distributed case, using Spark throughout. We'll cover Spark's programming model in detail, being careful to understand how and when it differs from familiar programming models, like shared-memory parallel collections or sequential Scala collections. Through hands-on examples in Spark and Scala, we'll learn when important issues related to distribution like latency and network communication should be considered and how they can be addressed effectively for improved performance.
Learning Outcomes. By the end of this course you will be able to:

- read data from persistent storage and load it into Apache Spark,
- manipulate data with Spark and Scala,
- express algorithms for data analysis in a functional style, 
- recognize how to avoid shuffles and recomputation in Spark,

Recommended background: You should have at least one year programming experience. Proficiency with Java or C# is ideal, but experience with other languages such as C/C++, Python, Javascript or Ruby is also sufficient. You should have some familiarity using the command line. This course is intended to be taken after Parallel Programming: https://www.coursera.org/learn/parprog1.

Note that this version of the course uses Scala 2.13. You can find a more recent version of the course that uses Scala 3 here: https://www.coursera.org/learn/scala-spark-big-data

Flexible deadlines

Reset deadlines in accordance to your schedule.

Shareable Certificate

Earn a Certificate upon completion

100% online

Start instantly and learn at your own schedule.

Intermediate Level

Approx. 27 hours to complete

English

Could your company benefit from training employees on in-demand skills?

Try Coursera for Business

Skills you will gain

Scala Programming
Big Data
Apache Spark
SQL

Flexible deadlines

Reset deadlines in accordance to your schedule.

Shareable Certificate

Earn a Certificate upon completion

100% online

Start instantly and learn at your own schedule.

Intermediate Level

Approx. 27 hours to complete

English

Could your company benefit from training employees on in-demand skills?

Try Coursera for Business

Instructor

Prof. Heather Miller

Assistant Professor

Carnegie Mellon University

96,601 Learners

2 Courses

Offered by

École Polytechnique Fédérale de Lausanne

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

Syllabus - What you will learn from this course

Week1

Week 1

12 hours to complete

Getting Started + Spark Basics

Get up and running with Scala on your computer. Complete an example assignment to familiarize yourself with our unique way of submitting assignments. In this week, we'll bridge the gap between data parallelism in the shared memory scenario (learned in the Parallel Programming course, prerequisite) and the distributed scenario. We'll look at important concerns that arise in distributed systems, like latency and failure. We'll go on to cover the basics of Spark, a functionally-oriented framework for big data processing in Scala. We'll end the first week by exercising what we learned about Spark by immediately getting our hands dirty analyzing a real-world data set.

12 hours to complete

7 videos (Total 105 min), 6 readings, 3 quizzes

7 videos

Introduction, Logistics, What You'll Learn15mData-Parallel to Distributed Data-Parallel10mLatency24mRDDs, Spark's Distributed Collection9mRDDs: Transformation and Actions16mEvaluation in Spark: Unlike Scala Collections!20mCluster Topology Matters!8m

6 readings

Tools Setup (Please read)10mSBT tutorial and Submission of Assignments (Please read)10mCheat Sheet10mScala Tutorial10mScala Style Guide10mWorking on Assignments5m

Week2

Week 2

7 hours to complete

Reduction Operations & Distributed Key-Value Pairs

7 hours to complete

4 videos (Total 59 min)

Week3

Week 3

1 hour to complete

Partitioning and Shuffling

1 hour to complete

4 videos (Total 57 min)

Week4

Week 4

8 hours to complete

Structured data: SQL, Dataframes, and Datasets

With our newfound understanding of the cost of data movement in a Spark job, and some experience optimizing jobs for data locality last week, this week we'll focus on how we can more easily achieve similar optimizations. Can structured data help us? We'll look at Spark SQL and its powerful optimizer which uses structure to apply impressive optimizations. We'll move on to cover DataFrames and Datasets, which give us a way to mix RDDs with the powerful automatic optimizations behind Spark SQL.

8 hours to complete

5 videos (Total 133 min)

Frequently Asked Questions

When will I have access to the lectures and assignments?
Access to lectures and assignments depends on your type of enrollment. If you take a course in audit mode, you will be able to see most course materials for free. To access graded assignments and to earn a Certificate, you will need to purchase the Certificate experience, during or after your audit. If you don't see the audit option:
The course may not offer an audit option. You can try a Free Trial instead, or apply for Financial Aid.
The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.
What will I get if I purchase the Certificate?
When you purchase a Certificate you get access to all course materials, including graded assignments. Upon completing the course, your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile. If you only want to read and view the course content, you can audit the course for free.
What is the refund policy?
You will be eligible for a full refund until two weeks after your payment date, or (for courses that have just launched) until two weeks after the first session of the course begins, whichever is later. You cannot receive a refund once you’ve earned a Course Certificate, even if you complete the course within the two-week refund period. See our full refund policy.
Is financial aid available?
Yes. In select learning programs, you can apply for financial aid or a scholarship if you can’t afford the enrollment fee. If fin aid or scholarship is available for your learning program selection, you’ll find a link to apply on the description page.

More questions? Visit the Learner Help Center.

Build employee skills, drive business results

Discover Coursera for Business

Big Data Analysis with Scala and Spark (Scala 2 version)

Big Data Analysis with Scala and Spark (Scala 2 version)

About this Course

Skills you will gain