Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
LLAP uses a completely separate read path in ORC to allow for caching and parallelization of reads and processing. This path does not support ACID. As far as I remember ACID logic is embedded inside ORC format; we need to refactor it to be on top of some interface, if practical; or just port it to LLAP read path.
Another consideration is how the logic will work with cache. The cache is currently low-level (CB-level in ORC), so we could just use it to read bases and deltas (deltas should be cached with higher priority) and merge as usual. We could also cache merged representation in future.
Attachments
Attachments
Issue Links
- blocks
-
HIVE-17470 eliminate potential vector copies when merging ACID deltas in LLAP IO path
- Open
-
HIVE-17458 VectorizedOrcAcidRowBatchReader doesn't handle 'original' files
- Closed
- is related to
-
HIVE-17916 remove ConfVars.HIVE_VECTORIZATION_ROW_IDENTIFIER_ENABLED
- Resolved
-
HIVE-12632 LLAP: don't use IO elevator for ACID tables
- Closed
- relates to
-
HIVE-17854 LlapRecordReader should have getRowNumber() like org.apache.orc.RecordReader
- Open
-
HIVE-15988 LLAP: cache ORC ACID delete deltas
- Resolved
- requires
-
HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions
- Resolved
- links to