This course is part of the Functional Programming in Scala Specialization

4.6
stars
2,574 ratings

Prof. Heather Miller

95,896 already enrolled

Offered By

Functional Programming in Scala SpecializationÉcole Polytechnique Fédérale de Lausanne

About this Course

20,471 recent views

Manipulating big data distributed over a cluster using functional concepts is rampant in industry, and is arguably one of the first widespread industrial uses of functional ideas. This is evidenced by the popularity of MapReduce and Hadoop, and most recently Apache Spark, a fast, in-memory distributed collections framework written in Scala. In this course, we'll see how the data parallel paradigm can be extended to the distributed case, using Spark throughout. We'll cover Spark's programming model in detail, being careful to understand how and when it differs from familiar programming models, like shared-memory parallel collections or sequential Scala collections. Through hands-on examples in Spark and Scala, we'll learn when important issues related to distribution like latency and network communication should be considered and how they can be addressed effectively for improved performance.
Learning Outcomes. By the end of this course you will be able to:

- read data from persistent storage and load it into Apache Spark,
- manipulate data with Spark and Scala,
- express algorithms for data analysis in a functional style, 
- recognize how to avoid shuffles and recomputation in Spark,

Recommended background: You should have at least one year programming experience. Proficiency with Java or C# is ideal, but experience with other languages such as C/C++, Python, Javascript or Ruby is also sufficient. You should have some familiarity using the command line. This course is intended to be taken after Parallel Programming: https://www.coursera.org/learn/parprog1.

Flexible deadlines

Reset deadlines in accordance to your schedule.

Shareable Certificate

Earn a Certificate upon completion

100% online

Start instantly and learn at your own schedule.

Course 4 of 5 in the

Functional Programming in Scala Specialization

Intermediate Level

Approx. 27 hours to complete

English

Could your company benefit from training employees on in-demand skills?

Try Coursera for Business

Skills you will gain

Scala Programming
Big Data
Apache Spark
SQL

Flexible deadlines

Reset deadlines in accordance to your schedule.

Shareable Certificate

Earn a Certificate upon completion

100% online

Start instantly and learn at your own schedule.

Course 4 of 5 in the

Functional Programming in Scala Specialization

Intermediate Level

Approx. 27 hours to complete

English

Could your company benefit from training employees on in-demand skills?

Try Coursera for Business

Instructor

Instructor rating

4.73/5 (145 Ratings)

Prof. Heather Miller

Assistant Professor

Carnegie Mellon University

96,601 Learners

2 Courses

Offered by

École Polytechnique Fédérale de Lausanne

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

Syllabus - What you will learn from this course

Content Rating92%(4,348 ratings)

Week1

Week 1

12 hours to complete

Getting Started + Spark Basics

Get up and running with Scala on your computer. Complete an example assignment to familiarize yourself with our unique way of submitting assignments. In this week, we'll bridge the gap between data parallelism in the shared memory scenario (learned in the Parallel Programming course, prerequisite) and the distributed scenario. We'll look at important concerns that arise in distributed systems, like latency and failure. We'll go on to cover the basics of Spark, a functionally-oriented framework for big data processing in Scala. We'll end the first week by exercising what we learned about Spark by immediately getting our hands dirty analyzing a real-world data set.

12 hours to complete

7 videos (Total 105 min), 7 readings, 3 quizzes

7 videos

Introduction, Logistics, What You'll Learn15mData-Parallel to Distributed Data-Parallel10mLatency24mRDDs, Spark's Distributed Collection9mRDDs: Transformation and Actions16mEvaluation in Spark: Unlike Scala Collections!20mCluster Topology Matters!8m

7 readings

Working on Assignments5mTools Setup (Please read)10mScala 3 REPL and Worksheets10mCheat Sheet10mSBT tutorial and Submission of Assignments (Please read)10mLearning Resources10mScala Style Guide10m

Week2

Week 2

7 hours to complete

Reduction Operations & Distributed Key-Value Pairs

7 hours to complete

4 videos (Total 59 min)

Week3

Week 3

1 hour to complete

Partitioning and Shuffling

1 hour to complete

4 videos (Total 57 min)

Week4

Week 4

8 hours to complete

Structured data: SQL, Dataframes, and Datasets

With our newfound understanding of the cost of data movement in a Spark job, and some experience optimizing jobs for data locality last week, this week we'll focus on how we can more easily achieve similar optimizations. Can structured data help us? We'll look at Spark SQL and its powerful optimizer which uses structure to apply impressive optimizations. We'll move on to cover DataFrames and Datasets, which give us a way to mix RDDs with the powerful automatic optimizations behind Spark SQL.

8 hours to complete

5 videos (Total 133 min)

Reviews

4.6

525 reviews

5 stars
73.04%
4 stars
21.08%
3 stars
4.38%
2 stars
0.66%
1 star
0.81%

TOP REVIEWS FROM BIG DATA ANALYSIS WITH SCALA AND SPARK

by SASep 22, 2019

Awesome course and awesome teacher! Nevertheless, to grasp the most of this course, you should do the previous 3 courses of the "Functional Programming in Scala" specialization.

by BLApr 2, 2020

some of the questions are unnecessarily specific (i.e. needs to be rounded to 1 decimal and sorted exactly for it to work)

but otherwise, great lecturer and great content

by NGMar 27, 2017

goot as introduction about spark and big data.

Small notice: it is incorrect to compare performance hadoop and spark. As I understand, spark was expected to be compacred with MapReduce.

by WDDec 2, 2017

Excellent course! It's clear the instructor put a ton of thought and hard work into this. I learned a lot that I wouldn't have learned without taking this class. Thank you, Heather!

View all reviews

About the Functional Programming in Scala Specialization

Discover how to write elegant code that works the first time it is run.

This Specialization provides a hands-on introduction to functional programming using the widespread programming language, Scala. It begins from the basic building blocks of the functional paradigm, first showing how to use these blocks to solve small problems, before building up to combining these concepts to architect larger functional programs. You'll see how the functional paradigm facilitates parallel and distributed programming, and through a series of hands on examples and programming assignments, you'll learn how to analyze data sets small to large; from parallel programming on multicore architectures, to distributed programming on a cluster using Apache Spark. A final capstone project will allow you to apply the skills you learned by building a large data-intensive application using real-world data.

Frequently Asked Questions

When will I have access to the lectures and assignments?
Access to lectures and assignments depends on your type of enrollment. If you take a course in audit mode, you will be able to see most course materials for free. To access graded assignments and to earn a Certificate, you will need to purchase the Certificate experience, during or after your audit. If you don't see the audit option:
The course may not offer an audit option. You can try a Free Trial instead, or apply for Financial Aid.
The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.
What will I get if I subscribe to this Specialization?
When you enroll in the course, you get access to all of the courses in the Specialization, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile. If you only want to read and view the course content, you can audit the course for free.
What is the refund policy?
If you subscribed, you get a 7-day free trial during which you can cancel at no penalty. After that, we don’t give refunds, but you can cancel your subscription at any time. See our full refund policy.
Is financial aid available?
Yes. In select learning programs, you can apply for financial aid or a scholarship if you can’t afford the enrollment fee. If fin aid or scholarship is available for your learning program selection, you’ll find a link to apply on the description page.

More questions? Visit the Learner Help Center.

Build employee skills, drive business results

Discover Coursera for Business

Big Data Analysis with Scala and Spark

About this Course

Skills you will gain