Details
-
Wish
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
After using MapReduce for many years, we have noticed that it lacks some important features: compound records, easy intra-reduce sorting and join capabilities. We have elaborated a slightly modified MapReduce foundation to overcome these problems: Tuple MapReduce. You can see a full paper published at the ICDM 2012 that describes it at http://pangool.net/TupleMapReduce.pdf
The good news are:
1) That no architectural changes on Hadoop are needed to embrace Tuple MapReduce.
2) Indeed, we have proven that it is possible to implement it on top of Hadoop. See the Pangool Open Source project ( http://pangool.net/ ).
3) It performs very efficiently ( http://pangool.net/benchmark.html )
4) It is compatible with all Hadoop stack: Writables, Serializers, Input/OutputFormats, etc.
We believe Hadoop community could benefit from it in different ways:
1) By getting ideas for a future API redesign.
2) By adopting Pangool inside Hadoop. Of course, we would be helping and contributing with anything needed doing any adaptation changes if needed (not many, because as I told, everything is compatible with existing MapReduce).
Obviously, we prefer the second. But at least, we believe some good ideas can be obtained by looking at Tuple MapReduce and Pangool.
There are also other improvements in Pangool that would improve Hadoop API:
1) Configuration by instance: passing parameters by constructor. For example, Pangool Input/OutputFormats can be configured by providing values to the constructor.
2) Stateful serialization. What is requested in https://issues.apache.org/jira/browse/MAPREDUCE-1462 is already supported by Pangool.
3) First-class multipleinput/multipleoutput.
Well, we are open to the discussion and to contribute.