[HIVE-1641] add map joined table to distributed cache

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.7.0
• Component/s: Query Processor
• Release Note: Split the MapJoin into 2 stage. In stage 1, generate the JDBM file for each small table. In stage 2, load the JDBM file and do the Join operation in memory
• Assignee: Liyin Tang
• Reporter: Namit Jain
• Created: 15/Sep/10 21:59
• Updated: 17/Dec/11 00:01
• Resolved: 26/Oct/10 18:29

Description

Currently, the mappers directly read the map-joined table from HDFS, which makes it difficult to scale.
We end up getting lots of timeouts once the number of mappers are beyond a few thousand, due to
concurrent mappers.

It would be good idea to put the mapped file into distributed cache and read from there instead.

Issue Links

• incorporates: HIVE-1722 The result of the test case mapjoin1.q is not correct
• incorporates: HIVE-1723 The result of left semi join is not correct

Sub-Tasks

• Keep the mapping between jdbm file name with bucket file name during the compile time: Resolved