Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27819

Retry cleanup of disk persisted RDD via external shuffle service when it failed via executor

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Spark Core
    • None

    Description

      This issue is created for not losing the idea came up during SPARK-27677 (at org.apache.spark.storage.BlockManagerMasterEndpoint#removeRdd):

      Quote

      In certain situations (e.g. executor death) it would make sense to retry this through the shuffle service. But I'm not sure how to safely detect those situations (or whether it makes sense to always retry through the shuffle service).

      Attachments

        Activity

          People

            Unassigned Unassigned
            attilapiros Attila Zsolt Piros
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: