[SPARK-50271] Spark App executes unintended commands

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.5.3
• Component/s: YARN
• Assignee: Unassigned
• Reporter: XianXianwang
• Created: 08/Nov/24 07:01
• Updated: 1 week ago 02:35

Description

Spark commits a task and Yarn executes the command directly using bash -c, if the commit contains ``` it will execute the statement directly, this has some impact on Yarn, does Saprk need to intercept this case

hadoop ContainerExecutor.java

ContainerLaunch.ShellScriptBuilder sb =
    ContainerLaunch.ShellScriptBuilder.create();
...
sb.command(command);

PrintStream pout = null;
try {
  pout = new PrintStream(out, false, "UTF-8");
  sb.write(pout);
} finally {
  if (out != null) {
    out.close();
  }
}