[HIVE-22731] MJ probe decode with row-level filtering

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: Hive, llap
• Labels: pull-request-available
• Assignee: Panagiotis Garefalakis Assign to me
• Reporter: Panagiotis Garefalakis
• Created: 15/Jan/20 16:56
• Updated: 25/Nov/20 15:58

Description

Currently, RecordReaders such as ORC support filtering at coarser-grained levels, namely: File, Stripe (64 to 256mb), and Row group (10k row) level. They only filter sets of rows if they can guarantee that none of the rows can pass a filter (usually given as searchable argument).

However, a significant amount of time can be spend decoding rows with multiple columns that are not even used in the final result. See figure where original is what happens today and in LazyDecode we skip decoding rows that do not match the key.

To enable a more fine-grained filtering in the particular case of a MapJoin we could utilize the key HashTable created from the smaller table to skip deserializing row columns at the larger table that do not match any key and thus save CPU time. 
This Jira investigates this direction. 

Issue Links

• depends upon: HIVE-23215 Make FilterContext and MutableFilterContext interfaces
• depends upon: ORC-577 Allow row-level filtering
• is blocked by: HIVE-23553 Upgrade ORC version to 1.6.7
• relates to: HIVE-23167 Expression probe decode with row-level filtering
• links to: GitHub Pull Request #926

Sub-Tasks

• Extend storage-api to expose FilterContext: Closed
• Basic compiler support for Probe MapJoin: Closed
• Compiler support tracking TS keyColName for Probe MapJoin: Closed
• Implement MJ HashTable contains key functionality: Closed
• Compiler extensions for MJ probe optimization: Closed
• Compiler probe MJ selection candidate fallback: Open
• [LLAP] propagate ProbeContex to LlapRecordReader: Closed
• [LLAP] support ColumnVectorBatch with FilterContext as part of ReadPipeline: Closed
• [LLAP] Probe-filter support for OrcEncodedDataConsumer: In Progress
• [LLAP] Extend InputFormat to genIncludedColNames: Closed
• Support multi-key probe MapJoins: Open