[BEAM-1197] Slowly-changing external data as a side input

• Type: Wish
• Status: Resolved
• Priority: P3
• Resolution: Duplicate
• Fix Version/s: Missing
• Component/s: beam-model
• Assignee: Unassigned
• Reporter: Eugene Kirpichov
• Created: 21/Dec/16 20:14
• Updated: 13/Jan/22 21:51
• Resolved: 23/Jul/21 20:48

Description

I've seen repeatedly the following pattern: a user wants to join a PCollection against a slowly-changing external dataset: e.g. a file on GCS, or a Bigtable, etc.

Side inputs come to mind, but current side input mechanisms don't allow for something like periodically reloading the side input.

The best hacky solution I came up with for one use case is documented here: http://stackoverflow.com/questions/41254028/can-dataflow-sideinput-be-updated-per-window-by-reading-a-gcs-bucket/41271159#41271159 , we need to do better than this.

Issue Links

• duplicates: BEAM-9650 Add consistent slowly changing side inputs support