[SPARK-43793] Fix SPARK_EXECUTOR_JAVA_OPTS assignment bug

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Spark Docker
• Assignee: Yikun Jiang
• Reporter: Yikun Jiang
• Created: 25/May/23 09:22
• Updated: 25/May/23 11:05
• Resolved: 25/May/23 11:05

Description

env | grep SPARK_JAVA_OPT_ |...

This is susceptible to a few bugs particularly around newlines in values. I see two ways around it.
ensuring the matching name is at the start, -E '^SPARK_JAVA_OPT_', and running all the commands with null-terminated input and output: env -0 and -z on the other commands
or bash variable prefix expansion

for v in "${!SPARK_JAVA_OPT_@}"; do
SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" )
done