[YARN-11346] Hardcoded absolute path for bash and mv command

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.4
• Component/s: nodemanager

Environment

Used Hadoop 3.3.4 downloaded from official site. Configs are in the attachment.

OS: NixOS-WSL (https://github.com/nix-community/NixOS-WSL)
JDK version: 8

• Assignee: Unassigned
• Reporter: wy
• Created: 13/Oct/22 06:40
• Updated: 02/Nov/22 02:44

Description

Not all operating systems have bash and mv in /bin folder. Seems that in HADOOP-13457 and HADOOP-13797 it is already aware but currently the issue still exists in hadoop-yarn.

For example, in NixOS, bash and mv is in /nix/store/..., linked to /run/current-system/sw/bin. In this case, when running:

./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.4.jar grep input output 'dfs[a-z.]+'

It will fail when running default_container_executor.sh with the following message:

Exception message: .../nm-local-dir/usercache/weko/appcache/application_1665637385308_0007/container_1665637385308_0007_02_000001/ .sh: line 2: /bin/bash: No such file or directory
.../nm-local-dir/usercache/weko/appcache/application_1665637385308_0007/container_1665637385308_0007_02_000001/default_container_executor.sh: line 5: /bin/mv: No such file or directory

because the script cannot find /bin/bash and /bin/mv.

For your information, "/usr/bin/env bash", which is more generic, can be used to replace "/bin/bash". It also works for "/bin/mv".