Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-3167

Update BigQueryIO to support schema auto detection when executing load jobs

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: io-java-gcp
    • Labels:
      None

      Description

      BigQuery supports detecting unspecified schemas automatically when executing load jobs: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load

      By default this is false and Dataflow does not set this to true when creating load jobs: https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L228

      Probably we should support specifying this as an option in BigQueryIO.

        Attachments

          Activity

            People

            • Assignee:
              chamikara Chamikara Jayalath
              Reporter:
              chamikara Chamikara Jayalath
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: