Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5497

Right anti, right outer and full outer hash joins sometimes do not flush resources early enough

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Backend

      Description

      This bug was found by running with IMPALA-4703 with the minimum possible reservation. The scenario is:

      1. Processing of the last output build partition finishes (in OutputUnmatchedBuild())
      2. Memory from that partition is attached to the output batch, bringing it to capacity.
      3. Execution continues, with more memory allocated in PrepareSpilledPartitionForProbe()

      The bug is that we should be flushing all memory before moving onto the next spilled partition - the code for processing a new spilled partition assumes that all of the reserved buffers are available.

            // Flush the remaining unmatched build rows of any partitions we are done
            // processing before moving onto the next partition.
            OutputUnmatchedBuild(out_batch);
            if (!output_build_partitions_.empty()) break;
      
            // Finished outputting unmatched build rows, move to next partition.
            DCHECK_EQ(builder_->num_hash_partitions(), 0);
            DCHECK(probe_hash_partitions_.empty());
            bool got_partition;
            RETURN_IF_ERROR(PrepareSpilledPartitionForProbe(state, &got_partition));
            if (!got_partition) {
              *eos = true;
              break;
            }
            if (out_batch->AtCapacity()) break;
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: