[BEAM-5049] Multiple batch joins on the same key results in multiple shuffles

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: dsl-sql
• Labels: starter
• Assignee: Unassigned
• Reporter: Anton Kedin
• Created: 31/Jul/18 17:23
• Updated: 03/Jun/22 22:13

Description

The query like this:

SELECT a.*, b.*, c.* FROM a JOIN b ON a.some_id = b.some_id JOIN c ON a.some_id = c.some_id;

results in two shuffles. Can probably be optimized.

Relevant code:

• BeamJoinRel implements Join in SQL: https://github.com/apache/beam/blob/1675b0f843ed34de8ba6f3676f794db80b40139d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java#L194

• CoGBK Join implementation: https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java#L36