Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25814

spark driver runs out of memory on org.apache.spark.util.kvstore.InMemoryStore

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

       We're looking into issue when even huge spark driver memory gets eventually exhausted and GC makes driver stop responding.

      Used JXRay.com tool and found that most of driver heap is used by 

       

      org.apache.spark.status.AppStatusStore
        -> org.apache.spark.status.ElementTrackingStore
          -> org.apache.spark.util.kvstore.InMemoryStore
       
      

       

      Is there is a way to tune this particular spark driver's memory region down?

       

       

       

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Tagar Ruslan Dautkhanov

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment