[SPARK-33005] Kubernetes GA Preparation

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.1
• Component/s: Kubernetes
• Labels: releasenotes
• Target Version/s: 3.1.0
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 26/Sep/20 23:21
• Updated: 12/Dec/22 18:10
• Resolved: 04/Jan/21 08:05

Issue Links

• relates to: SPARK-20624 SPIP: Add better handling for node shutdown
• relates to: SPARK-33461 Propagating SPARK_CONF_DIR in K8s and tests

Sub-Tasks

• Remove the words "experimental" in the k8s document: Resolved
• Support --proxy-user for Spark on K8s: Resolved
• Allow using some alternatives for service accounts: Resolved
• Allow setting serviceAccountName for executor pods: Resolved
• ExecutorPodsLifecycleManager has sub-optimal behavior with dynamic allocation: Resolved
• ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars: Resolved
• Driver cores in kubernetes are coupled with container resources, not spark.driver.cores: Resolved
• Fix flaky "Test basic decommissioning" test: Resolved
• Improve Spark Decommissioning K8s test relability: Resolved
• When processing new K8s state snapshots Spark treats Terminating nodes as terminated.: Resolved
• Decommissioning Integration Test checks for old string: Resolved
• Handle shuffle blocks being removed during decommissioning: Resolved
• Expose the ability for user to cleanup shuffle files: Resolved
• Support for Kubernetes NFS volume mounts: Resolved
• Support appId/execId placeholder in K8s SPARK_EXECUTOR_DIRS: Resolved
• Support execId placeholder in executor PVC conf: Resolved
• Support dynamic PVC creation/deletion for K8s executors: Resolved
• Support dynamic PVC creation/deletion for K8s driver: Resolved
• Remove Python2 test from K8s IT: Resolved
• Fix mismatched versions in test artifacts: Resolved
• Add dynamic PVC usage example into K8s doc: Resolved
• Fix typo of parameter in KubernetesTestComponents: Resolved
• Remove python2 installation in K8s python image: Resolved
• Use unique ConfigMap name for executor pod template: Resolved
• Improve error message for insufficient K8s volume confs: Resolved
• Upgrade fabric8 to 4.10.3: Resolved
• Fix & re-enable failing R K8s tests: Resolved
• Use LinkedHashMap instead of Map for newlyCreatedExecutors: Resolved
• Respect executor idle timeout conf in ExecutorPodsAllocator: Resolved
• Make spark.kubernetes.pyspark.pythonVersion allow `Python 3` only: Resolved
• Detect duplicated mountPath and fail at Spark side: Resolved
• Use 11-jre-slim as default in K8s Dockerfile: Resolved
• Use default Hadoop profile by removing explicit `-Phadoop-2.7` from K8s IT Jenkins job: Resolved
• Upgrade kubernetes-client to 4.11.1: Resolved
• Use R 3.6.3 in K8s R image: Resolved
• Spark client terminates while driver is still running: Resolved
• Spark executors on K8S do not request extra memory for off-heap allocations: Closed
• Kubernetes Configuration to set Service Account to Executors: Closed
• getMigrationBlocks should not fail at missing files: Resolved
• Support ordered shuffle block migration: Resolved
• Fix BlockManagerDecommissioner to return the correct migration status: Resolved
• Update misleading log messages for removed shuffle blocks: Resolved
• Add StorageLevel.DISK_ONLY_3: Resolved
• Upgrade kubernetes-client to 4.12.0: Resolved
• Support local dependency uploading from --py-files: Resolved
• Support Fallback Storage during Worker decommission: Resolved
• Make spark.archives working in Kubernates: Resolved
• Allow k8s integration tests to assert both driver and executor logs for expected log(s): Resolved
• Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables: Resolved
• Decommissioning Race Condition during Pod Snapshot: Resolved
• Update dynamic allocation related documents: Resolved
• Exclude fallback block manager from executorList: Resolved
• Fix fallback storage offset and improve compression codec test coverage: Resolved
• NPE in ExecutorPodsSnapshot: Resolved
• Race condition in Spark k8s Pod lifecycle manager that leads to shutdowns: Resolved
• Spark may report PodRunning if there is a sidecar that has not exited: Resolved
• Update kubernetes/integration-tests/README.md to follow the default Hadoop profile updated: Resolved
• Kubernetes integration tests doesn't work with Minikube 1.9+: Resolved
• `gpg: keyserver receive failed: No name` during K8s IT: Resolved
• Fix flaky test "Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties.": Resolved
• Support k8s cluster manager with stage level scheduling: Resolved
• Keep pending pods in account while scheduling new pods: Resolved
• Long runtime on many environment variables: Resolved
• Fix flaky test in DecommissionWorkerSuite: Resolved
• Promote spark.kubernetes.executor.podNamePrefix to the public conf: Resolved
• KubernetesClusterSchedulerBackend.stop should clean up K8s resources: Resolved
• ExecutorPodsAllocator fails to identify some excess requests during downscaling: Resolved
• K8s integration test should use the runtime Hadoop Version: Resolved
• Ignore RegisterExecutor when SparkContext is stopped: Resolved