[BEAM-3171] convert a join into lookup

• Type: New Feature
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: dsl-sql
• Labels: Done experimental
• Assignee: Mingmin Xu
• Reporter: Mingmin Xu
• Created: 10/Nov/17 18:41
• Updated: 24/Jul/20 20:20
• Resolved: 30/Jan/18 20:08

Description

We use BeamSQL to run streaming jobs mostly, and add a join_as_lookup improvement(internal branch) to cover the streaming-to-batch case(similar as [1]). I could submit a PR as experimental if people are interested.

The rough solution is, if one source of join node implements BeamSeekableTable and the other is not, then the join node is converted to a fact-lookup operation.

Ref:
[1] https://docs.google.com/document/d/1B-XnUwXh64lbswRieckU0BxtygSV58hysqZbpZmk03A/edit?usp=sharing

xumingming takidau for any comments

Issue Links

• links to: GitHub Pull Request #4196