Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-13990

BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API

Details

    • Improvement
    • Status: Open
    • P2
    • Resolution: Unresolved
    • 2.36.0
    • None
    • io-java-gcp

    Description

      when using Storage Write API with BigQueryIO, DATE and TIMESTAMP values are currently converted to String type in protobuf message. This is incorrect, according to storage write api documentation,] DATE should be converted to int32 and TIMESTAMP should be converted to int64.

      Here's error message: 

      INFO: Stream finished with error com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The proto field mismatched with BigQuery field at D6cbe536b_4dab_4292_8fda_ff2932dded49.datevalue, the proto field type string, BigQuery field type DATE Entity

      I have included an integration test here: https://github.com/liu-du/beam/commit/b56823d1d213adf6ca5564ce1d244cc4ae8f0816

       

      The problem is because DATE and TIMESTAMP are converted to String in protobuf message here: 

      https://github.com/apache/beam/blob/a78fec72d0d9198eef75144a7bdaf93ada5abf9b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L69

       

      Storage Write API reject the request because it's expecting int32/int64 values. 

       

      I've opened a PR here: https://github.com/apache/beam/pull/16926

      Attachments

        Activity

          People

            duliu Du Liu
            duliu Du Liu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 120h
                120h
                Remaining:
                Time Spent - 20h 40m Remaining Estimate - 99h 20m
                99h 20m
                Logged:
                Time Spent - 20h 40m Remaining Estimate - 99h 20m
                20h 40m