[SPARK-19609] Broadcast joins should pushdown join constraints as Filter to the larger relation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.1.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Nick Dimiduk
• Created: 15/Feb/17 18:59
• Updated: 12/Dec/22 18:10
• Resolved: 08/Oct/19 05:43

Description

For broadcast inner-joins, where the smaller relation is known to be small enough to materialize on a worker, the set of values for all join columns is known and fits in memory. Spark should translate these values into a Filter pushed down to the datasource. The common join condition of equality, i.e. lhs.a == rhs.a, can be written as an a in ... clause. An example of pushing such filters is already present in the form of IsNotNull filters via sameerag's work on SPARK-12957 subtasks.

This optimization could even work when the smaller relation does not fit entirely in memory. This could be done by partitioning the smaller relation into N pieces, applying this predicate pushdown for each piece, and unioning the results.

Issue Links

• is superceded by: SPARK-39753 Broadcast joins should pushdown join constraints as Filter to the larger relation
• relates to: SPARK-12957 Derive and propagate data constrains in logical plan