Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2128

Repartition instead of trying to allocate HT >1GB

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Works for Me
    • Affects Version/s: Impala 2.2
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      The current partitioned HJ and Agg implementation does not check what's the size of the hash table that it is going to be created. There are cases where the hash table of a partition to be needed to be larger than 1GB. In that case due to IMPALA-1619 we have to fail the query (for more info look the work-around IMPALA-2065).
      Instead of failing the query we may try to repartition that large partition in an effort to create HTs smaller than 1GB.

      For example in PartitionedHashJoinNode::Partition::BuildHashTableInternal

        int64_t estimated_num_buckets =
            HashTable::EstimateNumBuckets(build_rows()->num_rows());
      <-- Here we know whether the HT will be >1GB and should abort (goto not_built) -->
        hash_tbl_.reset(new HashTable(state, parent_->block_mgr_client_,
            parent_->child(1)->row_desc().tuple_descriptors().size(), build_rows(),
            1 << (32 - NUM_PARTITIONING_BITS), estimated_num_buckets));
      

        Attachments

          Activity

            People

            • Assignee:
              ippokratis Ippokratis Pandis
              Reporter:
              ippokratis Ippokratis Pandis
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: