Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25814

spark driver runs out of memory on org.apache.spark.util.kvstore.InMemoryStore

    Details

      Description

       We're looking into issue when even huge spark driver memory gets eventually exhausted and GC makes driver stop responding.

      Used JXRay.com tool and found that most of driver heap is used by 

       

      org.apache.spark.status.AppStatusStore
        -> org.apache.spark.status.ElementTrackingStore
          -> org.apache.spark.util.kvstore.InMemoryStore
       
      

       

      Is there is a way to tune this particular spark driver's memory region down?

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Tagar Ruslan Dautkhanov
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: