[YARN-10670] YARN: Opportunistic Container : : In distributed shell job if containers are killed then application is failed. But in this case as containers are killed to make room for guaranteed containers which is not correct to fail an application

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: distributed-shell
• Assignee: Bilwa S T
• Reporter: Sushanta Sen
• Created: 04/Mar/21 14:14
• Updated: 27/Apr/21 04:03

Description

Preconditions:

1. Secure Hadoop 3.1.1 - 3 Nodes cluster is installed
2. Set the below parameters  in RM yarn-site.xml ::<property>
   <name>yarn.resourcemanager.opportunistic-container-allocation.enabled</name>
   <value>true</value>
   </property>
3. Set this in NM[s]yarn-site.xml ::: <property>
   <name>yarn.nodemanager.opportunistic-containers-max-queue-length</name>
   <value>30</value>
   </property>

 
Test Steps:

Job Command : : yarn org.apache.hadoop.yarn.applications.distributedshell.Client jar HDFS/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1*.jar -shell_command sleep -shell_args 20 -num_containers 20 -container_type OPPORTUNISTIC -promote_opportunistic_after_start

Actual Result: Distributed Shell Yarn Job Failed with below Diagnostics message

Application Failure: desired = 20, completed = 20, allocated = 20, failed = 1, diagnostics = [2021-02-09 22:11:48.440]Container killed to make room for Guaranateed container.

 Expected Result: Distributed Shell Yarn Job should not fail.