[DRILL-6385] Support JPPD (Join Predicate Push Down)

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Server, Execution - Flow
• Labels: doc-impacting ready-to-commit
• Assignee: Weijie Tong
• Reporter: Weijie Tong
• Reviewer: Aman Sinha
• Created: 07/May/18 14:49
• Updated: 02/Sep/18 05:48
• Resolved: 23/Aug/18 08:15

Description

This feature is to support the JPPD (Join Predicate Push Down). It will benefit the HashJoin ,Broadcast HashJoin performance by reducing the number of rows to send across the network ,the memory consumed. This feature is already supported by Impala which calls it RuntimeFilter (https://www.cloudera.com/documentation/enterprise/5-9-x/topics/impala_runtime_filtering.html). The first PR will try to push down a bloom filter of HashJoin node to Parquet’s scan node.   The propose basic procedure is described as follow:

1. The HashJoin build side accumulate the equal join condition rows to construct a bloom filter. Then it sends out the bloom filter to the foreman node.
2. The foreman node accept the bloom filters passively from all the fragments that has the HashJoin operator. It then aggregates the bloom filters to form a global bloom filter.
3. The foreman node broadcasts the global bloom filter to all the probe side scan nodes which maybe already have send out partial data to the hash join nodes(currently the hash join node will prefetch one batch from both sides ).

      4.  The scan node accepts a global bloom filter from the foreman node. It will filter the rest rows satisfying the bloom filter.

 

To implement above execution flow, some main new notion described as below:

      1. RuntimeFilter

It’s a filter container which may contain BloomFilter or MinMaxFilter.

      2. RuntimeFilterReporter

It wraps the logic to send hash join’s bloom filter to the foreman.The serialized bloom filter will be sent out through the data tunnel.This object will be instanced by the FragmentExecutor and passed to the FragmentContext.So the HashJoin operator can obtain it through the FragmentContext.

     3. RuntimeFilterRequestHandler

It is responsible to accept a SendRuntimeFilterRequest RPC to strip the actual BloomFilter from the network. It then translates this filter to the WorkerBee’s new interface registerRuntimeFilter.

Another RPC type is BroadcastRuntimeFilterRequest. It will register the accepted global bloom filter to the WorkerBee by the registerRuntimeFilter method and then propagate to the FragmentContext through which the probe side scan node can fetch the aggregated bloom filter.

      4.RuntimeFilterManager

The foreman will instance a RuntimeFilterManager .It will indirectly get every RuntimeFilter by the WorkerBee. Once all the BloomFilters have been accepted and aggregated . It will broadcast the aggregated bloom filter to all the probe side scan nodes through the data tunnel by a BroadcastRuntimeFilterRequest RPC.

     5. RuntimeFilterEnableOption 

 A global option will be added to decide whether to enable this new feature.

 

Welcome suggestion and advice from you.The related PR will be presented as soon as possible.

Issue Links

• is related to: DRILL-6718 TPCH Queries hit IOB when running with the JPPD feature
• is related to: DRILL-6727 JPPD does not eliminate rows using the bloom filter if a HashJoin is involved
• links to: GitHub Pull Request #1334