[BEAM-13391] The Apache Avro library failed to parse the header with the following error: EOF reached

• Type: Bug
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.34.0
• Fix Version/s: 2.37.0
• Component/s: io-py-gcp

Language

Python

• Assignee: Sayat Satybaldiyev
• Reporter: Sayat Satybaldiyev
• Created: 06/Dec/21 23:25
• Updated: 19/Nov/22 11:31
• Resolved: 12/Jan/22 03:34

Description

When WriteToBigQuery is done with AVRO file loads, and the load is triggering a schema modification job, then following error is triggered:

E         RuntimeError: BigQuery job beam_bq_job_LOAD_AUTOMATIC_JOB_NAME_SCHEMA_MOD_STEP_171_736013176f6ea340cfd22b9f5b575c81_643a0ee608c04300bfbe67c8db85b2b0 failed. Error Result: <ErrorProto
E          message: 'Error while reading data, error message: The Apache Avro library failed to parse the header with the following error: EOF reached'
E          reason: 'invalid'> [while running 'WriteToBigQuery/BigQueryBatchFileLoads/WaitForSchemaModJobs']

Issue Links

• links to: GitHub Pull Request #16156