Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5967

JobProgressListener.stageIdToActiveJobIds never cleared

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 1.2.0, 1.2.1, 1.3.0
    • 1.2.2, 1.3.0
    • DStreams, Web UI
    • None

    Description

      The hashmap stageIdToActiveJobIds in JobProgressListener is never cleared. So the hashmap keep increasing in size indefinitely. This is a blocker for 24/7 running applications like Spark Streaming apps.

      Attachments

        Activity

          apachespark Apache Spark added a comment -

          User 'tdas' has created a pull request for this issue:
          https://github.com/apache/spark/pull/4741

          apachespark Apache Spark added a comment - User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/4741
          srowen Sean R. Owen added a comment -

          This might be a duplicate of this issue: https://issues.apache.org/jira/browse/SPARK-4906
          Not sure if it's related, but there's also: https://issues.apache.org/jira/browse/SPARK-3882

          srowen Sean R. Owen added a comment - This might be a duplicate of this issue: https://issues.apache.org/jira/browse/SPARK-4906 Not sure if it's related, but there's also: https://issues.apache.org/jira/browse/SPARK-3882
          tdas Tathagata Das added a comment -

          They are similar but not the same. Those were bugs in the old code, and this bug is in the new refactored JobProgressListener. This refactoring and hence the bug was done in Spark 1.2

          tdas Tathagata Das added a comment - They are similar but not the same. Those were bugs in the old code, and this bug is in the new refactored JobProgressListener. This refactoring and hence the bug was done in Spark 1.2
          srowen Sean R. Owen added a comment -

          tdas can one or both of those be resolved as obsolete then?

          srowen Sean R. Owen added a comment - tdas can one or both of those be resolved as obsolete then?

          People

            tdas Tathagata Das
            tdas Tathagata Das
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: