[SPARK-41550] Dynamic Allocation on K8S GA

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Kubernetes
• Labels: releasenotes
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 16/Dec/22 17:29
• Updated: 26/Jan/23 02:46
• Resolved: 26/Jan/23 02:46

Description

In the Kubernetes environments, this issue aims to make `Dynamic Allocation` as GA in Apache Spark 3.4.

Issue Links

• relates to: SPARK-37810 Executor Rolling in Kubernetes environment
• relates to: SPARK-41515 PVC-oriented executor pod allocation
• relates to: SPARK-20624 SPIP: Add better handling for node shutdown
• links to: Cloud Shuffle Storage Plugin for Apache Spark

Sub-Tasks

• Remove `Experimental` from `spark.dynamicAllocation.shuffleTracking.enabled`: Resolved
• Remove `Experimental` from `RDD.cleanShuffleDependencies` API: Resolved
• Remove `Dynamic Resource Allocation` from K8s Future Work: Resolved
• Enable spark.dynamicAllocation.shuffleTracking.enabled by default: Resolved
• Enable spark.storage.decommission.(rdd|shuffle)Blocks.enabled by default: Resolved
• Enable spark.storage.replication.proactive by default: Resolved
• Add PySpark cleanShuffleDependencies API: Resolved
• Reuse PVCs by default: Resolved
• ExecutorMonitor.timedOutExecutors should be deterministic: Resolved
• Log ExecutorDecommission as INFO level in TaskSchedulerImpl: Resolved
• Add decomTestTag to K8s Integration Test: Resolved
• Add `DISK_USED` executor roll policy: Resolved
• Add `TOTAL_SHUFFLE_WRITE` executor roll policy: Resolved
• Support PEAK_JVM_(ON|OFF)HEAP_MEMORY executor rolling policy: Resolved
• ExecutorMonitor.onExecutorRemoved should handle ExecutorDecommission as finished: Resolved
• Propagate decommission executor loss reason during onDisconnect in K8s: Resolved
• Keep removed executor info in decommission state: Resolved
• Populate ExecutorDecommission with more informative messages: Resolved
• Fix wrong remained shuffles log in BlockManagerDecommissioner: Resolved
• Ignore stage fetch failure caused by decommissioned executor: Resolved
• Randomize the orders of peer in BlockManagerDecommissioner: Resolved
• Add numberDecommissioningExecutors metric: Resolved
• FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname: Resolved
• Support `local[*]` in `spark-submit` in K8s environment: Resolved
• Log `exitCode` when `SparkContext.stop` starts: Resolved
• Ignore `IOException` in `handleBlockRemovalFailure` if SparkContext is stopped: Resolved