In this session, we look at the implementation of for-expressions. It turns out that there is a systematic translation scheme that maps for-expressions to certain combinations of higher order functions. >> So we're going to see in this session that the syntax of for-expressions is closely related to three higher-order functions, namely map, flatMap, and filter. First observation is that all these functions can be defined in terms of for. So for instance, you have a function like map, call it mapFun that applies a function f to every element on the list xs. We could write for (x taken from xs) yield f(x). That's the same thing that map is usually. FlatMap would be, for x taken from the first list and y taken from the result of applying f to each element of the first list, yield y. So that's a flap map. And filter finally can be expressed as a for like this. For x taken from xs if the predicate p is true at x, then yield x. So map, flat map and filter can all be expressed as for expressions. But in reality it goes the other way. What [INAUDIBLE] compiler will do is it will translate four expressions, two combinations of map, flatMap, and a lazy variant of filter. So I'm going to go to talk only about simple variables x on the left hand side of a generator where of course in general you could have an arbitrary pattern in this position. So let's look first at a simple for expression that consist of just one generator for x is taken from e1 yield e2 for [INAUDIBLE] expressions e1 and e2 so that would be translated by the Scala compiler to an application of map namely e1 map the function that takes an x and returns e2. The two are the same thing. Now that we've looked at for-expressions consisting only of a single generator, let's look at more complicated ones. The first for-expressions that we are going to look at would have a generator x taken from e1 followed by a filter f, and that in turn could be followed by further generators or Filters which are here substitution by s. That for expression can rewritten to the following one, another for expression that contains A generator and the filter has been absorbed into the generator. So the generator now reads x taken from e1 .withFilter, the anonymous function that takes an x and gives us back the expectation f. With filter, as a first approximation, you can read it like filter, so what happens here is that the generator reduced to all of those elements that pass the condition F. In fact with filter is a lazy variant of filter which means it doesn't immediately produce a new data structure of all the filtered elements, that would be wasteful What it does instead is, it remembers that any following call to map or flatmap has to be filtered by the function f. So it's essentially a smarter version of filter. The third and last form of for-expressions that we need to cover is the one where a leading generator is followed now not by a filter but By another generator and that in turn can be followed by an arbitrary sequence of filters and generators s. That for-expression will be translated into a call of flat map. SO the idea here is that if we take for-expression that takes All the remaining computations. So we generate a y from e2, then we do something more and then we yield e3. That would be a collection valued operation, because we have a generator here. And what we need to do is we need to take everything that comes out of this for expression and flatMap it. That means concatenate it all into the result list. That's precisely what happens here. So, we do it to an e1.flatMap with a function that takes an x and gives us back a for expression that has now one. Fewer generator. So it starts with the generator taken from e2 and s and then we get back e3. So what happened in the first case was a direct translation into a map. What happened in the second and third case was that we translated the For expression into one That has one less element. So it either has one fewer filter or one fewer generator. This one here had one fewer filter, the filter here has gone. Whereas this one here has a nested for expression, that now lacks the first generator. So each of these translation steps Can be repeated yielding simpler and simpler for expressions until finally, we must hit the simplest case and that then would translate into a map. So that's how a translation scheme works. Let's see it at As in our example. Let's take our for expression that looks at the pairs of indices whose sum is prime. So that was our for expression, if we apply our translation scheme mechanically We would be left with this for-expression here. So the first one here we have I until n. That's what you see here. It's a generator followed by a generator so I have a flatMap. The nested for-expression would take the generator 1 until i, that's a 1 here. That is followed by a filter, so we have the filter here, with a call to withFilter. And the call to this prime predicate. And finally, the whole thing gets mapped to a map where we form the paths. What's noteworthy, is that this is almost exactly the expression we came up with first. If you compare notes to what we had in the last session, you'll find only two differences. One was we use now with filter, instead of filter Which we expected all ready. It's just to save on allocations of intermediary data structures. And the second one is we do the filter slightly earlier. We do it as soon as we generate the second index. Where as before we did it once we had generated the pair. But that's just a minor transposition of operations. The end result of course, is exactly the same. So here's an exercise for you. Remember the queries we had on the books database. One of them was, give me the title of all books that have an author Author with a search of name. Your task is to take this query and translate it into a query that does not use a for expression, but only uses higher-order functions. So how would you solve that exercise? Well, let's follow the schema that we have seen previously, so we would start with books and we have two and we have two leading generators so that would lead to a flat map. Where our function takes the variable to the left of the generator. And we follow with a for-expression that contains the rest of the group before a followed by b.authors. If a starts with bird, Yield be the title. Now we have to take this second full expression here and translate it in terms. I will for the moment just write the result underneath. So what we see is it's a generator followed by a filter. So that means we pull the filter into the generator. So that would be for ( a taken from b.authors With filter. And then we take the a again, a=>a.startswith "Bird." And we yield, [INAUDIBLE] title. And remove the first one. So we still have a [INAUDIBLE] expression to translate. This one here now has a signal generator. So it would translate to a map. So let's do that. So we take the generator, and [INAUDIBLE] b.office With filter, a starts with bird. And now we take this one here, and map it. With the function y and return y.title. So that's the end result. It's a flatMap followed by a map of a generator that contains a with filter. Interestingly, the translation of for is not limited to just lists, or sequences, or even collections. All that we needed for the translation was the presence of methods map, flatMap and withFilter. Because that was what we translated into. So, that means that you can use the for expression syntax for your own types as well. All you need is to define map, flatMap, and withFilter for a type on which you want to you for expressions. And there are actually a lot of types for which this is useful. What comes to mind is kinds of collections such as arrays or iterators, databases, XML data. Optional values, for-expressions identifying to an option as well, parsers and many others as well. So, to stay with databases, for examples, books might not be a list or a set but a database stored in some server with data kept on disk. As long as the client interface to the database defines the methods map, flatMap and withFilter you can use for syntax for querying the database. And that's the basis of some popular Database connection frameworks for Scala which are called ScalaQuery and Slick. And similar ideas also underly Microsoft's LINQ, language integrated query framework, which let's your right expressions close to for expressions in C# versions of them are slighty different and map them to databases.
