Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17458

VectorizedOrcAcidRowBatchReader doesn't handle 'original' files

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.2.0
    • 3.0.0
    • Transactions
    • None

    Description

      VectorizedOrcAcidRowBatchReader will not be used for original files. This will likely look like a perf regression when converting a table from non-acid to acid until it runs through a major compaction.

      With Load Data support, if large files are added via Load Data, the read ops will not vectorize until major compaction.

      There is no reason why this should be the case. Just like OrcRawRecordMerger, VectorizedOrcAcidRowBatchReader can look at the other files in the logical tranche/bucket and calculate the offset for the RowBatch of the split. (Presumably getRecordReader().getRowNumber() works the same in vector mode).

      In this case we don't even need OrcSplit.isOriginal() - the reader can infer it from file path... which in particular simplifies OrcInputFormat.determineSplitStrategies()

      Attachments

        1. HIVE-17458.01.patch
          13 kB
          Eugene Koifman
        2. HIVE-17458.02.patch
          19 kB
          Eugene Koifman
        3. HIVE-17458.03.patch
          24 kB
          Eugene Koifman
        4. HIVE-17458.04.patch
          35 kB
          Eugene Koifman
        5. HIVE-17458.05.patch
          37 kB
          Eugene Koifman
        6. HIVE-17458.06.patch
          37 kB
          Eugene Koifman
        7. HIVE-17458.07.patch
          88 kB
          Eugene Koifman
        8. HIVE-17458.07.patch
          45 kB
          Eugene Koifman
        9. HIVE-17458.08.patch
          105 kB
          Eugene Koifman
        10. HIVE-17458.09.patch
          105 kB
          Eugene Koifman
        11. HIVE-17458.10.patch
          140 kB
          Eugene Koifman
        12. HIVE-17458.11.patch
          140 kB
          Eugene Koifman
        13. HIVE-17458.12.patch
          145 kB
          Eugene Koifman
        14. HIVE-17458.12.patch
          145 kB
          Eugene Koifman
        15. HIVE-17458.13.patch
          122 kB
          Eugene Koifman
        16. HIVE-17458.14.patch
          122 kB
          Eugene Koifman
        17. HIVE-17458.15.patch
          123 kB
          Eugene Koifman
        18. HIVE-17458.16.patch
          125 kB
          Eugene Koifman

        Issue Links

          Activity

            People

              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: