Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11015

fileIO issue on  latest version of apache-beam to 2.23.0

    XMLWordPrintableJSON

    Details

      Description

      In Apache Beam , while using below P transformation -fileio.MatchAll() and fileio.ReadMatches()  dataflow job on load performance .( processing 48 million resources)

      For 2-3 minute  Throughput(element/sec) goes to zero and no element was processed during that time , it comes back again back to average normal (0.25/sec) of my job

      In between these three minutes. There is no log from apache beam , if some IO error came or any warning.

      Please check for any fileIO  known issue on  latest version of apache-beam to 2.23.0

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              apoorva.e.sharma@accenture.com Apoorva Sharma
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: