[BEAM-7326] Document that Beam BigQuery IO expects users to pass base64-encoded bytes, and BQ IO serves base64-encoded bytes to the user.

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: io-java-gcp, io-py-gcp
• Assignee: Juta Staes
• Reporter: Valentyn Tymofieiev
• Created: 15/May/19 19:22
• Updated: 16/May/20 14:05
• Resolved: 28/Jun/19 21:41

Description

BYTES is one of the Datatypes supported by Google Cloud BigQuery, and Apache Beam BigQuery IO connector.

Current implementation of BigQuery connector in Java and Python SDKs expects that users base64-encode bytes before passing them to BigQuery IO, see discussion on dev: [1]

This needs to be reflected in public documentation, see [2-4]

cc: juta chamikara pabloem

cc: lostluck kedin FYI and to advise whether similar action needs to be done for Go SDK and/or Beam SQL.

[1] https://lists.apache.org/thread.html/f35c836887014e059527ed1a806e730321e2f9726164a3030575f455@%3Cdev.beam.apache.org%3E
[2] https://beam.apache.org/documentation/io/built-in/google-bigquery/
[3] https://beam.apache.org/releases/pydoc/2.12.0/apache_beam.io.gcp.bigquery.html
[4] https://beam.apache.org/releases/javadoc/2.12.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.html

Issue Links

• links to: GitHub Pull Request #8873