Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25814

spark driver runs out of memory on org.apache.spark.util.kvstore.InMemoryStore

    XMLWordPrintableJSON

Details

    Description

       We're looking into issue when even huge spark driver memory gets eventually exhausted and GC makes driver stop responding.

      Used JXRay.com tool and found that most of driver heap is used by 

       

      org.apache.spark.status.AppStatusStore
        -> org.apache.spark.status.ElementTrackingStore
          -> org.apache.spark.util.kvstore.InMemoryStore
       
      

       

      Is there is a way to tune this particular spark driver's memory region down?

       

       

       

      Attachments

        1. image-2018-10-23-14-06-53-722.png
          233 kB
          Ruslan Dautkhanov

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Tagar Ruslan Dautkhanov
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: