Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4236

External shuffle service must cleanup its shuffle files

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When external shuffle service is enabled, executors no longer cleanup their own shuffle files, as the external server can continue to serve them. The external serer must clean these files up once it knows that they will never be used again (i.e., when the application terminates or when Spark's ContextCleaner requests that they be deleted).

        Attachments

          Activity

            People

            • Assignee:
              ilikerps Aaron Davidson
              Reporter:
              ilikerps Aaron Davidson

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment