Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17327

NPE when starting MiniYARNCluster from hadoop-client-minicluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.3.0
    • 3.3.1, 3.4.0
    • None
    • None

    Description

      When starting MiniYARNCluster one could get the following exception:

        java.lang.NullPointerException:
        at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:72)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:122)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper.serviceStart(MiniYARNCluster.java:616)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:122)
        at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:327)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.spark.deploy.yarn.BaseYarnClusterSuite.beforeAll(BaseYarnClusterSuite.scala:96)
        ...
      

      Looking into the code, this is because we explicitly exclude resource files under hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL, and therefore this code in WebServer fails with NPE:

          terminalParams.put("resourceBase", WebServer.class
              .getClassLoader().getResource("TERMINAL").toExternalForm());
      

      Those who use hadoop-minicluster may not be affected because they'll also need hadoop-yarn-server-nodemanager as an extra dependency, which includes the resource files. On the other hand hadoop-client-minicluster packages both test classes (e.g., MiniYARNCluster) as well as main classes (e.g., ResourceManager and NodeManager) into a single shaded jar. It should include these resource files for testing as well. Otherwise, MiniYARNCluster is unusable.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              csun Chao Sun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: