Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38329

High I/O wait when Spark Structured Streaming checkpoint changed to EFS

    XMLWordPrintableJSON

Details

    Description

      I'm currently running spark structured streaming application written in python(pyspark) where my source is kafka topic and sink i mongodb. I changed my checkpoint to Amazon EFS, which is distributed on all spark workers and after that I got increased I/o wait, averaging 8%

       

      Currently I have 6000 messages coming to kafka every second, and I get every once in a while a WARN message:

      22/02/25 13:12:31 WARN HDFSBackedStateStoreProvider: Error cleaning up files for HDFSStateStoreProvider[id = (op=0,part=90),dir = file:/mnt/efs_max_io/spark/state/0/90] java.lang.NumberFormatException: For input string: ""

      I'm not quite sure if that message has anything to do with high I/O wait and is this behavior expected, or something to be concerned about?
       

      Attachments

        1. q.png
          23 kB
          Neven Jovic
        2. Screenshot from 2022-02-25 14-16-11.png
          30 kB
          Neven Jovic

        Activity

          People

            Unassigned Unassigned
            nevenj Neven Jovic
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: