Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5967

JobProgressListener.stageIdToActiveJobIds never cleared

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 1.2.0, 1.2.1, 1.3.0
    • 1.2.2, 1.3.0
    • DStreams, Web UI
    • None

    Description

      The hashmap stageIdToActiveJobIds in JobProgressListener is never cleared. So the hashmap keep increasing in size indefinitely. This is a blocker for 24/7 running applications like Spark Streaming apps.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tdas Tathagata Das
            tdas Tathagata Das
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment