Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8098

BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK correspond

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.17.0
    • Component/s: io-java-gcp
    • Labels:
      None

      Description

      While working on https://github.com/apache/beam/pull/9144, I realized there is a gap in BigQueryIO documentation on mapping between data types defined in BigQuery and in Beam SDK.

       

      For example, if a user reads a BYTES field from BigQuery into Beam, it will be represented as java.nio.ByteBuffer type in Beam Java SDK. The user will need to do an explicit type cast to ByteBuffer in order to use the data, but there is no easy way the user can know which type they should cast to, unless digging into BigQueryIO's implementation (Java - Avro - BigQuery).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                robinyqiu Robin Qiu
                Reporter:
                robinyqiu Robin Qiu
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m