[SPARK-45869] Revisit and Improve Spark Standalone Cluster

• Type: Epic
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: releasenotes
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 10/Nov/23 01:30
• Updated: 05/Sep/24 16:25
• Resolved: 10/Nov/23 02:35

Description

Spark Standalone Cluster has been supported for a long time as one of the resource managers.

As a part of Apache Spark 4.0.0, we revisit all layers of `Spark Standalone Cluster` as a long running subsystem inside K8s environment.

1. Spark Master, Worker, History Server Web UI Layer
2. Spark Master HA and Recovery Layer
3. Spark Master REST API Layer (including Cluster Utilization monitoring)
4. Spark Job Scheduling Layer
5. Spark Worker Management by exposing Cluster Utilization monitoring for Elastic Cluster Management
6. Spark Master/Worker dependency and classpath audit
7. Security
8. Documentation

Issue Links

• is related to: SPARK-45877 ExecutorFailureTracker support for standalone mode
• is related to: SPARK-45923 Spark Kubernetes Operator
• is related to: SPARK-44111 Prepare Apache Spark 4.0.0
• is related to: SPARK-47046 Apache Spark 4.0.0 Dependency Audit and Cleanup
• is related to: SPARK-49524 Improve K8s support

Sub-Tasks

• Add PersistenceEngineBenchmark: Resolved
• Include `Driver/App` data in `PersistenceEngineBenchmark`: Resolved
• Add RocksDBPersistenceEngine: Resolved
• Make RocksDBPersistenceEngine to support a symbolic link: Resolved
• Improve `PersistenceEngine` performance with `KryoSerializer`: Resolved
• Improve `FileSystemPersistenceEngine.persist` error message in case of the existing file: Resolved
• Improve `FileSystemPersistenceEngine` to allow non-exist parents: Resolved
• Improve `FileSystemPersistenceEngine` to support compressions: Resolved
• Improve `Master` to recover quickly in case of zero workers and apps: Resolved
• Enable `spark.worker.cleanup.enabled` by default: Resolved
• Support `spark.deploy.recoveryTimeout`: Resolved
• Make `spark.deploy.recovery*` documentation up-to-date: Resolved
• Support `JWSFilter`: Resolved
• Support `killall` in REST Submission API: Resolved
• Support `spark.master.rest.host`: Resolved
• Support `clear` in REST Submission API: Resolved
• Support `readyz` in REST Submission API: Resolved
• Support server-side `environmentVariables` replacement in REST Submission API: Resolved
• Support server-side `sparkProperties` replacement in REST Submission API: Resolved
• Make `appArgs` and `environmentVariables` optional in REST API: Resolved
• Make `spark.app.name` property optional in REST API: Resolved
• Support `spark.deploy.maxDrivers`: Resolved
• Support `spark.deploy.spreadOutDrivers`: Resolved
• Support `spark.deploy.workerSelectionPolicy`: Resolved
• Support `spark.worker.idPattern`: Resolved
• Support `spark.deploy.driverIdPattern`: Resolved
• Support `spark.deploy.appIdPattern`: Resolved
• Support `spark.deploy.appNumberModulo` to rotate app number: Resolved
• Support `spark.master.useAppNameAsAppId.enabled`: Resolved
• Support `spark.master.useDriverIdAsAppName.enabled`: Resolved
• Support `spark.test.appId` in `LocalSchedulerBackend`: Resolved
• Support `spark.master.ui.historyServerUrl` in `ApplicationPage`: Resolved
• Support `spark.worker.(initial|max)RegistrationRetries`: Resolved
• Support `spark.driver.timeout` and `DriverTimeoutPlugin`: Resolved
• Support `spark.master.rest.filters`: Resolved
• Support Spark Master Log UI: Resolved
• Support Spark Worker Log UI: Resolved
• Support Spark History Server Log UI: Resolved
• Support Spark Driver Live Log UI: Resolved
• Support top-level filtering in MasterPage JSON API: Resolved
• Add `Environment` page to Master UI: Resolved
• Add `Environment Variables` table to Master `EnvironmentPage`: Resolved
• Improve `MasterPage` to support custom title: Resolved
• Support custom History Server UI title: Resolved
• Show a summary of workers in MasterPage: Resolved
• Show the number of drivers waiting in SUBMITTED status: Resolved
• Show the number of abnormally completed drivers in MasterPage: Resolved
• Show `Duration` in `ApplicationPage`: Resolved
• Improve `MasterPage` to show `Resource` column only when it exists: Resolved
• Show driver log location in Spark History Server: Resolved
• Show the number of cached RDDs in StoragePage: Resolved
• Hide `Thread Dump` and `Heap Histogram` of `Dead` executors in `Executors` UI: Resolved
• Make StandaloneRestServer add JavaModuleOptions to drivers: Resolved
• Fix WorkerPage to use the same pattern for `logPage` urls: Resolved
• Fix getBaseURI error in Spark Worker LogPage UI buttons: Resolved
• Fix `MasterPage` to sort `Running Drivers` table by `Duration` column correctly: Resolved
• Fix Spark History Server to sort `Duration` column properly: Resolved
• Collect and update `spark-standalone.md` with new confs: Resolved
• Fix `Spark Standalone` documentation table layout: Resolved
• Make single-pod spark jobs respect spark.app.id: Resolved
• Document `spark.master.*` configurations: Resolved
• EventLogFileReader should not read rolling logs if appStatus is missing: Resolved
• Redact `awsAccessKeyId` by including `accesskey` pattern: Resolved
• Log the final state of drivers during `Master.removeDriver`: Resolved
• Log Spark HA recovery duration: Resolved
• Warn properly when a driver exists successfully but master is disconnected: Resolved
• Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message: Resolved
• Remove `kill` link from RELAUNCHING drivers in MasterPage: Resolved
• Remove `*slave*.sh` scripts: Resolved
• Refactor to improve `RegisterWorker` unit test: Resolved
• Rename spark.deploy.spreadOut to spark.deploy.spreadOutApps: Resolved
• Improve `MasterSuite` to use nanoTime-based appIDs and workerIDs: Resolved
• Fix `spark-daemon.sh` usage by adding `decommission` command: Resolved
• Make `WorkerResourceInfo` extend `Serializable` explicitly: Resolved
• Add `logrotate` to Spark docker files: Resolved
• Recover `log-view.js` to be non-module: Resolved
• Support `/json/clusterutilization` API: Resolved
• Fix `Master` to reject worker kill request if decommission is disabled: Resolved
• Ensure trailing slashes in `HistoryServer` URL redirections: Resolved
• Validate `spark.master.ui.decommission.allow.mode` setting: Resolved
• Remove POST APIs from `MasterWebUI` when spark.ui.killEnabled is false: Resolved
• Fix `Load New` button in `Master/HistoryServer` Log UI: Resolved
• Check logType in Utils.getLog: Resolved
• Use `getTotalMemorySize` in `WorkerArguments`: Resolved
• Document REST API for Spark Standalone Cluster: Resolved
• Document `SPARK_LOG_*` and `SPARK_PID_DIR`: Resolved
• Document spark.network.timeoutInterval: Resolved
• Document Spark Driver Live Log UI: Resolved
• Document MasterPage custom title conf and REST API server-side env variable replacements: Resolved
• Document `JWSFilter` usage in Spark UI and REST API and rename parameter to `secretKey`: Resolved
• Update `Configuring Ports for Network Security` section with JWS: Resolved
• Fix `RealBrowserUISeleniumSuite`: Resolved
• Add `WebBrowserTest`: Resolved
• Exclude `CodeHaus Jackson` dependencies from Master/Worker/HistoryServer classpaths: Resolved
• Fix `MasterSuite` to validate the number of registered workers: Resolved
• Reduce the number of required threads in MasterSuite: Resolved
• Make `PluginEndpoint` warn when plugins reply for one-way message: Resolved
• Make `BlockManager` warn before `removeBlockInternal`: Resolved
• Add `jjwt` profile: Resolved
• Add `submit_pi.sh` REST API example: Resolved
• Add `submit-sql.sh` REST API example: Resolved
• Move `spark.history.ui.maxApplications` config definition to `History.scala`: Resolved
• Redact `Spark Command` output in `launcher` module: Resolved
• Make Spark Deamons support `spark.log.structuredLogging.enabled`: Resolved
• Spark deamons should respect spark.log.structuredLogging.enabled conf: Resolved
• Add JavaSparkSQLCli example: Resolved
• Simplify the log when Spark HybridStore hits the memory limit: Resolved
• Fix `ApplicationPage` to hide App UI link when UI is disabled: Resolved
• Fix 'MasterPage' to hide App UI links when UI is disabled: Resolved
• Unify `*.sh` file naming patterns by replacing `_` with `-`: Resolved
• Fix `StandaloneRestServer` to propagate `spark.app.name` to SparkSubmit properly: Resolved
• Fix `SparkSubmit` to show REST API `kill` response properly: Resolved
• Support `spark.submit.appName`: Resolved
• Define `BLOCK_MANAGER_REREGISTRATION_FAILED` as `ExecutorExitCode`: Resolved