Details
-
Improvement
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
2.2.0
-
None
Description
VectorizedOrcAcidRowBatchReader will not be used for original files. This will likely look like a perf regression when converting a table from non-acid to acid until it runs through a major compaction.
With Load Data support, if large files are added via Load Data, the read ops will not vectorize until major compaction.
There is no reason why this should be the case. Just like OrcRawRecordMerger, VectorizedOrcAcidRowBatchReader can look at the other files in the logical tranche/bucket and calculate the offset for the RowBatch of the split. (Presumably getRecordReader().getRowNumber() works the same in vector mode).
In this case we don't even need OrcSplit.isOriginal() - the reader can infer it from file path... which in particular simplifies OrcInputFormat.determineSplitStrategies()
Attachments
Attachments
Issue Links
- is blocked by
-
HIVE-17923 'cluster by' should not be needed for a bucketed table
- Resolved
-
HIVE-12631 LLAP IO: support ORC ACID tables
- Closed
-
HIVE-14233 Improve vectorization for ACID by eliminating row-by-row stitching
- Resolved
- is related to
-
HIVE-14878 integrate MM tables into ACID: add separate ACID type
- Resolved
-
HIVE-17854 LlapRecordReader should have getRowNumber() like org.apache.orc.RecordReader
- Open
- relates to
-
HIVE-16812 VectorizedOrcAcidRowBatchReader doesn't filter delete events
- Closed
-
HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions
- Resolved
-
HIVE-18045 can VectorizedOrcAcidRowBatchReader be used all the time
- Open
- links to