Map-Reduce and its Applications

MapReduce Features and Architectures
Features: interface:
- simplified, abstracted and powerful
- restricted
hardware:
- commodity(low-cost, unreliable) machines
system:
- extremely scalable RAIN cluster
- high performance (parallel,)
- fault-tolerance and simple administration

Architecture: Shared-nothing computing and disk-shared storage
There are a global file system (say, GFS) and a cluster of computing nodes each of which has a local processor, memory and hard drive. To run a job with a dataset, mapreduce starts with reading splitted dataset from GFS and assigning each trunk to a node in the cluster. A comstomable map function is then called for each trunk, which outputs an immediate file containing a series of key-value pairs (with duplicated and sporadic keys). Then, a partition-sort-group operation is performed to generate inputs for reduce operations, which essentially aggragates all values associated with the same key. In the end, a series of key-value pairs (with distinct keys or values) are generated as the final results. The procedure could be illustrated as follows.

read (k1, v1) from FS and schedule them to a cluster of nodes
map: (k1, v1) -> list(k2, v2) {duplicated and sporadic k2}
partition-sort-group: list(k2, v2) -> (k2, list(v2)) {distinct k2}
reduce: (k2, list(v2)) -> list(v3) {distinct v3}

Note the initial schedule and manipulation of intermediate files (partition-sort-group operation) are performed by general MapReduce framework.

Observation & Motivation
What programmer want for data analysis?
- procedural programming: so pure sql is out.
- high-level declarative primitives (say, sql): so Map-Reduce interface is out, so we have Pig Latin.

What is present now?
- Map-Reduce framework:
-- the interface is rigid, thereby limiting programming power, (say fail to support n-stage data flow functions, like join).
-- too low level. programmers need to build tasks from scratch, even for the common tasks.

Technical Problems
Map-Reduce fails to support sql primitives, mainly join (over heterogeneous data sets).


FQ:
why shared-nothing means scalability?
why simply map-reduce can't treat well with data-join?
what is differences between P2P and cloud computing?(control in cloud computing is centralized)