The final thing is locality.
Um, so a cloud, uh, typically has a hierarchical topology.
For instance, a cloud might have multiple racks,
and communication inside a rack is typically much faster
than across racks, which goes across the core switch.
Uh, the file system that is underlying, h-uh,
MapReduce or Hadoop, which might be GFS or HDFS,
typically stores three replicas of each of the chunks or blocks.
For instance, the blocks might be 64 megabytes in size.
Uh, typically these are stored on two different racks.
There might be two stored on one rack,
two replicas of the block stored on one rack,
and one stored on a different rack.
These are stored on two different racks
so that if one of the racks fails, for instance,
the top of the rack switch goes down,
then you at least have one copy of the file around, uh,
or-or the block around.
So given this, uh, MapReduce or Hadoop attempts to schedule
a Map task on preferably a machine or server
that contains a replica of the corresponding input data,
so that, uh, the input to that task,
this is typically a Map task,
uh, would, uh, be, uh, uh, fetched
without incurring any network overhead.
However, this may not be possible
because all the servers on which-all the three servers
on which this Map input is located
might already have all their containers, um, uh, full,
meaning they're already running tasks.
In this case, the scheduler tries to schedule, uh,
the Map task on the same rack
as a server that contains the input.
Uh, this has the advantage that the inter, uh, the intrarack,
or the inside rack bandwidth can be used,
which is typically much faster than going across racks,
in order to transfer, uh, the input block,
and this is going to be fairly fast,
not as fast as running on the same machine
as where the data is,
but faster than going across, uh, the core switch.
However, this may not be possible either,
because all the servers or all the racks where, uh,
this input block is located may be using up
all of their containers.
In this case, uh, the, uh, MapReduce scheduler
would then schedule the task anywhere
where a free container is available.
So essentially, uh, these three
are, uh, orders of, uh, preferring
where to schedule a Map task.
For scheduling Reduce tasks
really, um, you don't have much of a choice
because you don't really know up front
where the data from these Reduce tasks is gonna be coming from.
Remember the data for a Reduce task comes from one
or potentially all of the Map tasks in that job.
Uh, and so typically you want to schedule the Reduce tasks
on the same racks as where the Map tasks are scheduled uh,
but in terms of which machines
you want to prefer in those racks, uh,
you may want to prefer at least those machines
where Map tasks are scheduled
but you, uh, really don't know much about
uh, which particular Map tasks
you want to collocate your Reduce tasks with.