[HIVE-15221] Improvement for MapJoin checkMemoryStatus, adding gc before throwing Exception

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.1, 2.1.0
• Component/s: Query Processor
• Target Version/s: 3.0.0
• Assignee: Hui Fei
• Reporter: Hui Fei
• Created: 16/Nov/16 07:22
• Updated: 01/Oct/19 22:07

Description

i see in the current master version

MapJoinMemoryExhaustionHandler.java

  public void checkMemoryStatus(long tableContainerSize, long numRows)
  throws MapJoinMemoryExhaustionException {
    long usedMemory = memoryMXBean.getHeapMemoryUsage().getUsed();
    double percentage = (double) usedMemory / (double) maxHeapSize;
    String msg = Utilities.now() + "\tProcessing rows:\t" + numRows + "\tHashtable size:\t"
        + tableContainerSize + "\tMemory usage:\t" + usedMemory + "\tpercentage:\t" + percentageNumberFormat.format(percentage);
    console.printInfo(msg);
    if(percentage > maxMemoryUsage) {
      throw new MapJoinMemoryExhaustionException(msg);
    }
   }

if percentage > maxMemoryUsage, then throw MapJoinMemoryExhaustionException

in my opinion, running is better than fail. after System.gc, ' if percentage > maxMemoryUsage, then throw MapJoinMemoryExhaustionException' maybe better

And original checking way has a problem: 1) consuming much memory cause gc (e.g young gc), then check after adding row and pass. 2) consuming much memory does not cause gc, then check after adding rows but throw Exception

sometimes 2) occurs, but it contians less rows than 1).