Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4236

External shuffle service must cleanup its shuffle files

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When external shuffle service is enabled, executors no longer cleanup their own shuffle files, as the external server can continue to serve them. The external serer must clean these files up once it knows that they will never be used again (i.e., when the application terminates or when Spark's ContextCleaner requests that they be deleted).

        Attachments

          Activity

            People

            • Assignee:
              ilikerps Aaron Davidson
              Reporter:
              ilikerps Aaron Davidson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: