[YARN-11709] NodeManager should be shut down or blacklisted when it cannot run program "/var/lib/yarn-ce/bin/container-executor"

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.5.0
• Component/s: container-executor
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Benjamin Teke
• Reporter: Ferenc Erdelyi
• Created: 25/Jul/24 08:24
• Updated: 17/Sep/24 12:24
• Resolved: 17/Sep/24 12:24

Description

When NodeManager encounters the below "No such file or directory" error reported against the "container-executor", it should give up participating in the cluster as it is not capable to run any container, but just fail the jobs.

2023-01-18 10:08:10,600 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container container_e159_1673543180101_9407_02_
000014 startLocalizer is : -1
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: java.io.IOException: Cannot run program "/var/lib/yarn-ce/bin/container-executor": error=2, No such file or directory
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:183)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:403)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.j
ava:1250)
Caused by: java.io.IOException: Cannot run program "/var/lib/yarn-ce/bin/container-executor": error=2, No such file or directory

Issue Links

• blocks: YARN-11715 NodeManager should recover by itself once the container-executor can run program again
• causes: YARN-11721 Do not mark the NM unhealthy when an app is killed
• links to: GitHub Pull Request #6960
• links to: GitHub Pull Request #6992
• links to: GitHub Pull Request #7028
• links to: GitHub Pull Request #7043