Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.7.0
-
None
-
Split the MapJoin into 2 stage. In stage 1, generate the JDBM file for each small table. In stage 2, load the JDBM file and do the Join operation in memory
Description
Currently, the mappers directly read the map-joined table from HDFS, which makes it difficult to scale.
We end up getting lots of timeouts once the number of mappers are beyond a few thousand, due to
concurrent mappers.
It would be good idea to put the mapped file into distributed cache and read from there instead.