[HIVE-15789] Vectorization: limit reduce vectorization to 32Mb chunks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: Vectorization
• Target Version/s: 2.2.0
• Release Note: Vectorization: limit reduce vectorization to 32Mb chunks (Teddy Choi, via Gopal V)
• Assignee: Teddy Choi
• Reporter: Gopal Vijayaraghavan
• Created: 02/Feb/17 03:48
• Updated: 21/Jul/17 18:36
• Resolved: 22/Mar/17 21:03

Description

Reduce vectorization accumulates 1024 rows before forwarding it into the reduce processor.

Add a safety limit for 32Mb of writables, so that shorter sequences can be forwarded into the operator trees.

        rowIdx++;
        if (rowIdx >= BATCH_SIZE) {
          VectorizedBatchUtil.setBatchSize(batch, rowIdx);
          reducer.process(batch, tag);

Issue Links

• Dependent: HIVE-15700 BytesColumnVector can get stuck trying to resize byte buffer
• links to: Review Board