[IMPALA-5497] Right anti, right outer and full outer hash joins sometimes do not flush resources early enough

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.10.0
• Epic Color: ghx-label-8
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 13/Jun/17 16:01
• Updated: 21/Jul/17 06:18
• Resolved: 21/Jun/17 20:59

Description

This bug was found by running with IMPALA-4703 with the minimum possible reservation. The scenario is:

1. Processing of the last output build partition finishes (in OutputUnmatchedBuild())
2. Memory from that partition is attached to the output batch, bringing it to capacity.
3. Execution continues, with more memory allocated in PrepareSpilledPartitionForProbe()

The bug is that we should be flushing all memory before moving onto the next spilled partition - the code for processing a new spilled partition assumes that all of the reserved buffers are available.

      // Flush the remaining unmatched build rows of any partitions we are done
      // processing before moving onto the next partition.
      OutputUnmatchedBuild(out_batch);
      if (!output_build_partitions_.empty()) break;

      // Finished outputting unmatched build rows, move to next partition.
      DCHECK_EQ(builder_->num_hash_partitions(), 0);
      DCHECK(probe_hash_partitions_.empty());
      bool got_partition;
      RETURN_IF_ERROR(PrepareSpilledPartitionForProbe(state, &got_partition));
      if (!got_partition) {
        *eos = true;
        break;
      }
      if (out_batch->AtCapacity()) break;

Issue Links

• is depended upon by: IMPALA-4703 Add reservation stress option for test coverage