[HIVE-10793] Hybrid Hybrid Grace Hash Join : Don't allocate all hash table memory upfront

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: Hive
• Labels: TODOC1.3
• Assignee: Mostafa Mokhtar
• Reporter: Mostafa Mokhtar
• Created: 22/May/15 04:09
• Updated: 16/Feb/16 23:50
• Resolved: 26/May/15 19:25

Description

HybridHashTableContainer will allocate memory based on estimate, which means if the actual is less than the estimate the allocated memory won't be used.

Number of partitions is calculated based on estimated data size

numPartitions = calcNumPartitions(memoryThreshold, estimatedTableSize, minNumParts, minWbSize,
          nwayConf);

Then based on number of partitions writeBufferSize is set

writeBufferSize = (int)(estimatedTableSize / numPartitions);

Each hash partition will allocate 1 WriteBuffer, with no further allocation if the estimate data size is correct.

Suggested solution is to reduce writeBufferSize by a factor such that only X% of the memory is preallocated.