Hive
  1. Hive
  2. HIVE-6276

batchSize computation in Vectorize ORC reader can cause BufferUnderFlowException when PPD is enabled

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.13.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      nextBatch() method that computes the batchSize is only aware of stripe boundaries. This will not work when PPD in ORC is enabled as PPD works at row group level (stripe contains multiple row groups). By default, row group stride is 10000. When PPD is enabled, some row groups may get eliminated. After row group elimination, disk ranges are computed based on the selected row groups. If batchSize computation is not aware of this, it will lead to BufferUnderFlowException (reading beyond disk range). Following scenario should illustrate it more clearly

      |--------------------------------- STRIPE 1 ------------------------------------|
      |-- row grp 1 --|-- row grp 2 --|-- row grp 3 --|-- row grp 4 --|-- row grp 5 --|
                      |--------- diskrange 1 ---------|               |- diskrange 2 -|
                                                      ^
                                                   (marker)   
      

      diskrange1 will have 20000 rows and diskrange 2 will have 10000 rows. Since nextBatch() was not aware of row groups and hence the diskranges, it tries to read 1024 values from the end of diskrange 1 where it should only read 20000 % 1024 = 544 values. This will result in BufferUnderFlowException.

      To fix this, a marker is placed at the end of each range and batchSize is computed accordingly.

      batchSize = Math.min(VectorizedRowBatch.DEFAULT_SIZE, (markerPosition - rowInStripe));

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        22h 28m 1 Prasanth Jayachandran 24/Jan/14 02:38
        Prasanth Jayachandran made changes -
        Field Original Value New Value
        Resolution Duplicate [ 3 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Hide
        Prasanth Jayachandran added a comment -

        Duplicate of HIVE-6287. Got created as JIRA was flaky yesterday.

        Show
        Prasanth Jayachandran added a comment - Duplicate of HIVE-6287 . Got created as JIRA was flaky yesterday.
        Prasanth Jayachandran created issue -

          People

          • Assignee:
            Prasanth Jayachandran
            Reporter:
            Prasanth Jayachandran
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development