In Apache Beam , while using below P transformation -fileio.MatchAll() and fileio.ReadMatches() dataflow job on load performance .( processing 48 million resources)
For 2-3 minute Throughput(element/sec) goes to zero and no element was processed during that time , it comes back again back to average normal (0.25/sec) of my job
In between these three minutes. There is no log from apache beam , if some IO error came or any warning.
Please check for any fileIO known issue on latest version of apache-beam to 2.23.0