Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-384

Streaming BigQueryIO should support user-provided IDs

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.1.0-incubating, 0.2.0-incubating
    • Fix Version/s: None
    • Component/s: io-java-gcp
    • Labels:
      None

      Description

      Currently, BigQueryIO always assigns IDs and does a shuffle to ensure that they are atomic. This incurs a noticeable cost and is unnecessary if the user already has deterministic IDs that they can use. The sink should be able to use these IDs to skip the shuffle.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              millsd@google.com Daniel Mills
            • Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: