[SPARK-49984] Duplicate JVM options

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.3
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Adam Binford
• Created: 16/Oct/24 01:15
• Updated: 18/Oct/24 15:15

Description

JVM options in spark.driver.defaultJavaOptions and spark.executor.defaultJavaOptions end up being added multiple times to the command line due to the way the Java module options and IPv6 options are added.

$ pyspark --conf spark.driver.defaultJavaOptions="-Dfoo=bar"

>>> spark.conf.get('spark.driver.extraJavaOptions')

'-Djava.net.preferIPv6Addresses=false -Dfoo=bar -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dfoo=bar'

>>> spark.conf.get('spark.driver.defaultJavaOptions')

'-Dfoo=bar'

Issue Links

• links to: GitHub Pull Request #48488