Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.3.0
-
None
-
Running Spark on K8S with supplied Docker image. Passing along extra java options.
Description
`spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` when processed in `entrypoint.sh` does not preserve its ordering, which makes `-XX:+UnlockExperimentalVMOptions` unusable, as you have to pass it before any other experimental options.
Steps to reproduce:
- Set `spark.driver.extraJavaOptions`, e.g. `-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -XX:+UseCGroupMemoryLimitForHeap`
- Submit application to k8s cluster.
- Fetch logs and observe that on each run order of options is different and when `-XX:+UnlockExperimentalVMOptions` is not the first startup will fail.
Expected behaviour:
- Order of `extraJavaOptions` should be preserved.
Cause:
`entrypoint.sh` fetches environment options with `env`, which doesn't guarantee ordering.
env | grep SPARK_JAVA_OPT_ | sed 's/[^=]*=\(.*\)/\1/g' > /tmp/java_opts.txt