[BEAM-12729] Suppress Avro Runtime Exceptions for Streaming

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Component/s: io-java-avro
• Labels: streaming
• Flags: Patch

Language

java

• Assignee: Unassigned
• Reporter: Dylan Hercher
• Created: 06/Aug/21 22:40
• Updated: 14/Jan/22 00:48
• Resolved: 14/Jan/22 00:48

Description

The current design of ReadFileRangesFn continually throws any un-recoverable errors until the pipeline is turned down for streaming pipelines.

 

These invalid file errors do not have a resolvable solution and should be logged as errors for the files in question to allow the pipeline to continue progressing.

 

As a file which cannot be read will never recover via a dead letter queue design.  Since no recovery is possible we can simply log the errored file and continue processing.

Issue Links

• links to: GitHub Pull Request #15298