Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27677

Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.3
    • Fix Version/s: 3.0.0
    • Component/s: Block Manager, Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      Disk-cached RDD blocks are currently unavailable after an executor is removed. However, when there is an external shuffle service, the data remains available on disk and could be served by the shuffle service. This would allow dynamic allocation to reclaim executors with only disk-cached blocks more rapidly, but still keep the cached data available.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                attilapiros Attila Zsolt Piros
                Reporter:
                irashid Imran Rashid
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: