[BEAM-11015] fileIO issue on latest version of apache-beam to 2.23.0

• Type: Bug
• Status: Triage Needed
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.23.0
• Fix Version/s: Not applicable
• Component/s: runner-dataflow, sdk-java-core
• Assignee: Unassigned
• Reporter: Apoorva Sharma
• Created: 05/Oct/20 13:20
• Updated: 26/Jan/21 17:35

Description

In Apache Beam , while using below P transformation -fileio.MatchAll() and fileio.ReadMatches()  dataflow job on load performance .( processing 48 million resources)

For 2-3 minute  Throughput(element/sec) goes to zero and no element was processed during that time , it comes back again back to average normal (0.25/sec) of my job

In between these three minutes. There is no log from apache beam , if some IO error came or any warning.

Please check for any fileIO  known issue on  latest version of apache-beam to 2.23.0