Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15789

Vectorization: limit reduce vectorization to 32Mb chunks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.3.0
    • Vectorization
    • None
    • Vectorization: limit reduce vectorization to 32Mb chunks (Teddy Choi, via Gopal V)

    Description

      Reduce vectorization accumulates 1024 rows before forwarding it into the reduce processor.

      Add a safety limit for 32Mb of writables, so that shorter sequences can be forwarded into the operator trees.

              rowIdx++;
              if (rowIdx >= BATCH_SIZE) {
                VectorizedBatchUtil.setBatchSize(batch, rowIdx);
                reducer.process(batch, tag);
      

      Attachments

        1. HIVE-15789.1.patch
          4 kB
          Teddy Choi
        2. HIVE-15789.2.patch
          4 kB
          Teddy Choi

        Issue Links

          Activity

            People

              teddy.choi Teddy Choi
              gopalv Gopal Vijayaraghavan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: