[HADOOP-17327] NPE when starting MiniYARNCluster from hadoop-client-minicluster

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 24/Oct/20 00:16
• Updated: 11/Feb/24 13:13
• Resolved: 11/Nov/20 07:13

Description

When starting MiniYARNCluster one could get the following exception:

  java.lang.NullPointerException:
  at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:72)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:122)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper.serviceStart(MiniYARNCluster.java:616)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:122)
  at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:327)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.spark.deploy.yarn.BaseYarnClusterSuite.beforeAll(BaseYarnClusterSuite.scala:96)
  ...

Looking into the code, this is because we explicitly exclude resource files under hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL, and therefore this code in WebServer fails with NPE:

    terminalParams.put("resourceBase", WebServer.class
        .getClassLoader().getResource("TERMINAL").toExternalForm());

Those who use hadoop-minicluster may not be affected because they'll also need hadoop-yarn-server-nodemanager as an extra dependency, which includes the resource files. On the other hand hadoop-client-minicluster packages both test classes (e.g., MiniYARNCluster) as well as main classes (e.g., ResourceManager and NodeManager) into a single shaded jar. It should include these resource files for testing as well. Otherwise, MiniYARNCluster is unusable.

Issue Links

• is broken by: YARN-8914 Add xtermjs to YARN UI2
• is duplicated by: HADOOP-17324 Don't relocate org.bouncycastle in shaded client jars