[HIVE-16425] Vectorization: unload old hashtables before reloadHashTable

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Vectorization
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 12/Apr/17 11:14
• Updated: 22/May/18 23:58
• Resolved: 12/Apr/17 17:44

Description

  @Override
  protected void reloadHashTable(byte pos, int partitionId)
          throws IOException, HiveException, SerDeException, ClassNotFoundException {

    // The super method will reload a hash table partition of one of the small tables.
    // Currently, for native vector map join it will only be one small table.
    super.reloadHashTable(pos, partitionId);

    MapJoinTableContainer smallTable = spilledMapJoinTables[pos];

    vectorMapJoinHashTable = VectorMapJoinOptimizedCreateHashTable.createHashTable(conf,
        smallTable);
    needHashTableSetup = true;
    LOG.info("Created " + vectorMapJoinHashTable.getClass().getSimpleName() + " from " + this.getClass().getSimpleName());

    if (isLogDebugEnabled) {
      LOG.debug(CLASS_NAME + " reloadHashTable!");
    }
  }

The super call causes an OOM because of existing memory usage by vectorMapJoinHashTable.