Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8393

Java BigQueryIO clustering support breaks on multiple partitions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P1
    • Resolution: Fixed
    • Affects Version/s: 2.15.0, 2.16.0
    • Fix Version/s: 2.17.0
    • Component/s: io-java-gcp
    • Labels:
      None

      Description

      Support for writing to clustered tables in BigQuery was added in 2.15, which involved adding a new optional clustering field to TableDestination. Clustering support is working for most cases, but fails with errors about incompatible partitioning specifications for any data that is handled by the MultiplePartitions branch of BigQueryIO logic.

      There is a case in that code path where we provide a modified TableDestination and neglect to copy the clustering definition, so the final load job does not include any clustering columns.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jeff.klukas@gmail.com Jeff Klukas
                Reporter:
                jeff.klukas@gmail.com Jeff Klukas
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m