[BEAM-12765] Cannot access field fruit on a value with type ARRAY<STRUCT<fruit STRING>>

• Type: Bug
• Status: Resolved
• Priority: P0
• Resolution: Fixed
• Affects Version/s: 2.33.0
• Fix Version/s: 2.33.0
• Component/s: sdk-py-core, test-failures
• Labels: currently-failing
• Assignee: Pablo Estrada
• Reporter: Luke Cwik
• Created: 16/Aug/21 22:18
• Updated: 19/Aug/21 17:05
• Resolved: 19/Aug/21 17:05

Description

I believe https://github.com/apache/beam/pull/15237 is the root cause.

Example failure:
https://ci-beam.apache.org/job/beam_PostCommit_Python38/1564/testReport/junit/apache_beam.io.gcp.bigquery_write_it_test/BigQueryWriteIntegrationTests/test_big_query_write_temp_table_append_schema_update/

Tests are failing with:

google.api_core.exceptions.BadRequest: 400 Cannot access field fruit on a value with type ARRAY<STRUCT<fruit STRING>> at [1:53]  (job ID: 3dcd4354-fcae-44dd-8e27-cff6ad41155c)                                                    -----Query Job SQL Follows-----                                                         |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    1:SELECT bytes, date, time, int64, bool, nested_field.fruit FROM python_write_to_table_16291383366439.python_append_schema_update     |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |

Issue Links

• links to: Github PR #15352
• links to: GitHub Pull Request #15337