[HIVE-17458] VectorizedOrcAcidRowBatchReader doesn't handle 'original' files

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 3.0.0
• Component/s: Transactions
• Target Version/s: 3.0.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 06/Sep/17 01:22
• Updated: 22/May/18 23:16
• Resolved: 04/Nov/17 17:15

Description

VectorizedOrcAcidRowBatchReader will not be used for original files. This will likely look like a perf regression when converting a table from non-acid to acid until it runs through a major compaction.

With Load Data support, if large files are added via Load Data, the read ops will not vectorize until major compaction.

There is no reason why this should be the case. Just like OrcRawRecordMerger, VectorizedOrcAcidRowBatchReader can look at the other files in the logical tranche/bucket and calculate the offset for the RowBatch of the split. (Presumably getRecordReader().getRowNumber() works the same in vector mode).

In this case we don't even need OrcSplit.isOriginal() - the reader can infer it from file path... which in particular simplifies OrcInputFormat.determineSplitStrategies()

Issue Links

• is blocked by: HIVE-17923 'cluster by' should not be needed for a bucketed table
• is blocked by: HIVE-12631 LLAP IO: support ORC ACID tables
• is blocked by: HIVE-14233 Improve vectorization for ACID by eliminating row-by-row stitching
• is related to: HIVE-14878 integrate MM tables into ACID: add separate ACID type
• is related to: HIVE-17854 LlapRecordReader should have getRowNumber() like org.apache.orc.RecordReader
• relates to: HIVE-16812 VectorizedOrcAcidRowBatchReader doesn't filter delete events
• relates to: HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions
• relates to: HIVE-18045 can VectorizedOrcAcidRowBatchReader be used all the time
• links to: Review Board

Sub-Tasks

• can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done?: Closed
• Enable VectorizedOrcAcidRowBatchReader to be used with LLAP IO elevator over original acid files: In Progress
• VectorizedOrcAcidRowBatchReader.computeOffsetAndBucket optimization: Closed
• Aggregation with struct in LLAP produces wrong result: Closed
• Enable runWorker() UDF to launch compactor from .q tests: Open
• select ROW__ID, t, si, i from over10k_orc_bucketed where b = 4294967363 and t < 100 order by ROW__ID fails on LLAP: Closed
• OrcSplit.canUseLlapIo() disables LLAP IO for non-vectorized acid reads: Resolved
• clean up acid_vectorization_original.q: Closed