[SPARK-27677] Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.3
• Fix Version/s: 3.0.0
• Component/s: Block Manager, Spark Core
• Target Version/s: 3.0.0
• Assignee: Attila Zsolt Piros
• Reporter: Imran Rashid
• Created: 10/May/19 21:40
• Updated: 06/Aug/21 17:07
• Resolved: 23/May/19 20:16

Description

Disk-cached RDD blocks are currently unavailable after an executor is removed. However, when there is an external shuffle service, the data remains available on disk and could be served by the shuffle service. This would allow dynamic allocation to reclaim executors with only disk-cached blocks more rapidly, but still keep the cached data available.

Issue Links

• Blocked: SPARK-36446 YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor
• relates to: SPARK-25888 Service requests for persist() blocks via external service after dynamic deallocation
• links to: [Github] Pull Request #24499 (attilapiros)
• links to: [Github] Pull Request #24499 (attilapiros)
• links to: GitHub Pull Request #24499
• links to: GitHub Pull Request #24697