[HIVE-11467] WriteBuffers rounding wbSize to next power of 2 may cause OOM

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0, 2.0.0
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: Hive
• Assignee: Wei Zheng
• Reporter: Wei Zheng
• Created: 05/Aug/15 18:04
• Updated: 16/Feb/16 23:53
• Resolved: 13/Aug/15 21:47

Description

If wbSize passed to WriteBuffers cstr is not power of 2, it will do a rounding first to the next power of 2

  public WriteBuffers(int wbSize, long maxSize) {
    this.wbSize = Integer.bitCount(wbSize) == 1 ? wbSize : (Integer.highestOneBit(wbSize) << 1);
    this.wbSizeLog2 = 31 - Integer.numberOfLeadingZeros(this.wbSize);
    this.offsetMask = this.wbSize - 1;
    this.maxSize = maxSize;
    writePos.bufferIndex = -1;
    nextBufferToWrite();
  }

That may break existing memory consumption assumption for mapjoin, and potentially cause OOM.

The solution will be to pass a power of 2 number as wbSize from upstream during hashtable creation, to avoid this late expansion.