Declarative Languages

Loading...

From the course by University of Washington

Data Manipulation at Scale: Systems and Algorithms

650 ratings

University of Washington

Data Manipulation at Scale: Systems and Algorithms

650 ratings

Course 1 of 4 in the Specialization Data Science at Scale

Data analysis has replaced data acquisition as the bottleneck to evidence-based decision making --- we are drowning in it. Extracting knowledge from large, heterogeneous, and noisy datasets requires not only powerful computing resources, but the programming abstractions to use them effectively. The abstractions that emerged in the last decade blend ideas from parallel databases, distributed systems, and programming languages to create a new class of scalable data analytics platforms that form the foundation for data science at realistic scales. In this course, you will learn the landscape of relevant systems, the principles on which they rely, their tradeoffs, and how to evaluate their utility against your requirements. You will learn how practical systems were derived from the frontier of research in computer science and what systems are coming on the horizon. Cloud computing, SQL and NoSQL databases, MapReduce and the ecosystem it spawned, Spark and its contemporaries, and specialized systems for graphs and arrays will be covered. You will also learn the history and context of data science, the skills, challenges, and methodologies the term implies, and how to structure a data science project. At the end of this course, you will be able to: Learning Goals: 1. Describe common patterns, challenges, and approaches associated with data science projects, and what makes them different from projects in related fields. 2. Identify and use the programming models associated with scalable data manipulation, including relational algebra, mapreduce, and other data flow models. 3. Use database technology adapted for large-scale analytics, including the concepts driving parallel databases, parallel query processing, and in-database analytics 4. Evaluate key-value stores and NoSQL systems, describe their tradeoffs with comparable systems, the details of important examples in the space, and future trends. 5. “Think” in MapReduce to effectively write algorithms for systems including Hadoop and Spark. You will understand their limitations, design details, their relationship to databases, and their associated ecosystem of algorithms, extensions, and languages. write programs in Spark 6. Describe the landscape of specialized Big Data systems for graphs, arrays, and streams

From the lesson

Relational Databases and the Relational Algebra

Relational Databases are the workhouse of large-scale data management. Although originally motivated by problems in enterprise operations, they have proven remarkably capable for analytics as well. But most importantly, the principles underlying relational databases are universal in managing, manipulating, and analyzing data at scale. Even as the landscape of large-scale data systems has expanded dramatically in the last decade, relational models and languages have remained a unifying concept. For working with large-scale data, there is no more important programming model to learn.

Optimization: Physical Query Plans5:29

Optimization: Choosing Physical Plans4:54

Declarative Languages5:24

Declarative Languages: More Examples4:53

Views: Logical Data Independence5:07

Meet the Instructors

Bill Howe
Director of Research
Scalable Data Analytics

[MUSIC]

Okay, last time we talked about algebraic optimization.

And I argued that all three of these expressions,

without going into a lot of detail.

But I argued that all three of these were equivalent and

they differed only in the order in which things were evaluated.

Here you evaluate this join first and this join second.

And in this expression you evaluate this join first and this join second, and here

you sort of find all possible combinations of tuples and then filter that.

So if you don't understand exactly what's going on in these expressions that's okay,

you're not going to know that yet.

We'll talk about it, in fact, in this segment I think, but

the takeaway here is that there is three equivalent expressions and

we don't necessarily which one is the fastest one to evaluate?

But the database can figure this out, and does every time you write a query okay.

And that's this notion of algebraic optimization.

We don't, even if you're familiar with databases you may or

may not be familiar with relational algebra, which should be strange because

I've argued that it's the hallmark of databases and totally fundamental.

So, why don't we think about programming

databases in terms of writing relational algebra expressions?

Well, another good idea, another key idea that's associated with a relational

database is this notion of declarative languages.

What we mean by declarative languages is that you specify the answer that you want,

but you do not specify anything about how to get it.

And so a relational algebra expression

actually does specify an order, right, as I showed on this slide.

Here's three different expressions that's indicating

exactly which order to do every operation.

That means that if you write an expression like this,

you're instructing the computer, look do it in this particular order.

So these declarative languages say look,

we're just going to describe the properties that must be true of the result

and we're gonna let the database figure out the right order in which to do this.

So here's a quick example, so

imagine you have two tables, one is order with three columns, order, date and

account and another table with item with two columns, order and part.

The semantics here is that this

column indicates which order that item should be associated with okay.

And so, if you want to say find all orders from today along with the items

ordered, then you may write this query and

if you've seen SQL plenty of times before, bear with me.

And if you haven't, then pay attention.

So select star give me all possible columns from the table order and

all possible columns from the table item But

I only want record such that this condition is true, where the order column

from the order table matches the order column from the item table right?

And, further, I only want orders from today where o.date = today().

So this is just conditions expressed over the results

without any kind of idea of how to actually get this answer.

So what automatically happens is that this query is translated into a relational

algebra expression along the lines of what we've already seen.

You know, here I've sort of just done a cartoon where you can say,

scan the item table, scan the order table,

select the record such that date equals today and then perform the join.

Find all the records in order that correspond to, for each record in

order find the corresponding records and item that match on order.

Okay?

So, this is happening every time you run a query, again.

So, the SQL is the what, not the how.

Given another example, there's three columns, product, purchase and customer.

And the underlying in here we haven't talked about, but

this is indicating what makes this record unique.

And so here, the PID make the product unique.

The CID makes the customer unique and the combination of PID and

CID makes the purchase unique okay.

So here's another SQL query, we say, select distinct product name.

Why do I know it's the product name?

Cuz I see an x here and I see an x here.

And, the customer name, and I know it's the customer name cuz I see z here and

I see a z here.

This is an alias for the relation customer.

From these two or three tables, Where the product ID in the product

table matches the product ID in the purchase table and the customer ID and

the purchase table matches the customer ID in the customer table.

And that's a typo, it looks like that should be z.

And then we want, but now we want only the products for which the price is greater

than 100, and we only want the customers whose city is Seattle.

All right, so what does this say in English?

Well, if I need combinations of products and

customers, you need print combinations of products and customers where the customer

is in Seattle, and they pay for a product worth more than a 100 okay?

So it's clear what we want but it's unclear how to get it.

It's kind of a complicated query.

[MUSIC]

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2018 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play