[BEAM-8098] BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK correspond

• Type: Improvement
• Status: Triage Needed
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.17.0
• Component/s: io-java-gcp
• Assignee: Robin Qiu
• Reporter: Robin Qiu
• Created: 27/Aug/19 05:46
• Updated: 13/Apr/23 10:57
• Resolved: 12/Sep/19 21:34

Description

While working on https://github.com/apache/beam/pull/9144, I realized there is a gap in BigQueryIO documentation on mapping between data types defined in BigQuery and in Beam SDK.

 

For example, if a user reads a BYTES field from BigQuery into Beam, it will be represented as java.nio.ByteBuffer type in Beam Java SDK. The user will need to do an explicit type cast to ByteBuffer in order to use the data, but there is no easy way the user can know which type they should cast to, unless digging into BigQueryIO's implementation (Java - Avro - BigQuery).

Issue Links

• links to: GitHub Pull Request #9555