Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5144

HashTableSink allocates empty new Object[] arrays & OOMs - use a static emptyRow instead

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Query Processor
    • Labels:
    • Environment:

      Ubuntu LXC + -Xmx512m client opts

    • Release Note:
      Save memory in HashTableSink for key-only map-joins, by reusing a single EMPTY_ROW_CONTAINER object

      Description

      The map-join hashtable sink in the local-task creates an in-memory hashtable with the following code.

       Object[] value = JoinUtil.computeMapJoinValues(row, joinValues[alias],
      ...
       MapJoinRowContainer rowContainer = tableContainer.get(key);
          if (rowContainer == null) {
            rowContainer = new MapJoinRowContainer();
            rowContainer.add(value);
      

      But for a query where the joinValues[alias].size() == 0, this results in a large number of unnecessary allocations which would be better served with a copy-on-write default value container & a pre-allocated zero object array which is immutable (the only immutable array there is in java).

      The query tested is roughly the following to scan all of customer_demographics in the hash-sink

      select c_salutation, count(1)
       from customer
            JOIN customer_demographics ON customer.c_current_cdemo_sk = customer_demographics.cd_demo_sk
       group by c_salutation
       limit 10
      ;
      
      

      When running with current trunk, the code results in an OOM with 512Mb ram.

      2013-08-23 05:11:26	Processing rows:	1400000	Hashtable size:	1399999	Memory usage:	292418944	percentage:	0.579
      
      Execution failed with exit status: 3
      Obtaining error information
      

        Attachments

        1. HIVE-5144.02.patch
          2 kB
          Gopal Vijayaraghavan
        2. HIVE-5144.01.patch
          2 kB
          Gopal Vijayaraghavan

          Activity

            People

            • Assignee:
              gopalv Gopal Vijayaraghavan
              Reporter:
              gopalv Gopal Vijayaraghavan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: