Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24786

Executors not being released after all cached data is unpersisted



    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 2.3.0
    • None
    • Scheduler, Spark Core
    • None
    • Zeppelin in EMR


      I'm persisting a dataframe in Zeppelin which has dynamic allocation enabled to get a sense of how much memory the dataframe takes up. After I note the size, I unpersist the dataframe. For some reason, Yarn is not releasing the executors that were added to Zeppelin. If I don't run the persist and unpersist steps, the executors that were added are removed about a minute after the paragraphs complete. Looking at the storage tab in the Spark UI for the Zeppelin job, I don't see anything cached. I do not want to set spark.dynamicAllocation.cachedExecutorIdleTimeout to a lower value because I do not want executors with cached data to be released, but I do want ones that had cached data and no longer have cached data to be released.


      Steps to reproduce:

      1. Enable dynamic allocation
      2. Set spark.dynamicAllocation.executorIdleTimeout to 60s
      3. Set spark.dynamicAllocation.cachedExecutorIdleTimeout to infinity
      4. Load a dataset, persist it, run a count on the persisted dataset, unpersist the persisted dataset
      5. Wait a couple minutes

      Expected behaviour:

      All executors will be released as the executors are no longer caching any data

      Observed behaviour:

      No executors were released


        Issue Links



              Unassigned Unassigned
              jcharles Jeffrey Charles
              0 Vote for this issue
              6 Start watching this issue