Hadoop Common
  1. Hadoop Common
  2. HADOOP-309

NullPointerException in StatusHttpServer

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.10.1
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      The NullPointerException happens in the constructor of StatusHttpServer because the System property "hadoop.log.dir" is undefined.
      I see it trying to start the Namenode directly without using any scripts.

      Exception in thread "main" java.lang.NullPointerException
      at org.mortbay.util.Resource.newResource(Resource.java:99)
      at org.mortbay.http.ResourceCache.setResourceBase(ResourceCache.java:140)
      at org.mortbay.http.HttpContext.setResourceBase(HttpContext.java:2207)
      at org.apache.hadoop.mapred.StatusHttpServer.<init>(StatusHttpServer.java:66)
      at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:172)
      at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:97)
      at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:88)
      at org.apache.hadoop.dfs.NameNode.main(NameNode.java:496)

      In general I think the sources should not rely on the system properties and environment variables defined in hadoop scripts.

      1. 309.patch
        1 kB
        navychen

        Activity

        Hide
        Konstantin Shvachko added a comment -

        Even if I get through this I get yet another NullPointerException.
        Now because webapps resource is not defined.

        Exception in thread "main" java.lang.NullPointerException
        at org.apache.hadoop.mapred.StatusHttpServer.getWebAppsPath(StatusHttpServer.java:108)
        at org.apache.hadoop.mapred.StatusHttpServer.<init>(StatusHttpServer.java:71)
        at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:172)
        at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:97)
        at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:88)
        at org.apache.hadoop.dfs.NameNode.main(NameNode.java:496)

        Show
        Konstantin Shvachko added a comment - Even if I get through this I get yet another NullPointerException. Now because webapps resource is not defined. Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.mapred.StatusHttpServer.getWebAppsPath(StatusHttpServer.java:108) at org.apache.hadoop.mapred.StatusHttpServer.<init>(StatusHttpServer.java:71) at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:172) at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:97) at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:88) at org.apache.hadoop.dfs.NameNode.main(NameNode.java:496)
        Hide
        Owen O'Malley added a comment -

        As we discussed offline, this is just caused because you didn't define a value for "hadoop.log.dir" on your java command line. It would be good to have the server not crash when the property is not defined, but the property should be defined for the servers. The two obvious "fixes" are to:
        1. Remove the "/logs" context when no "hadoop.log.dir" property is defined. This will be confusing to users.
        2. See if we can get the value from log4j via the jakarta commons logging api. It looks like there might be routines that will work.

        In any case, I believe this is a pretty low priority bug.

        Show
        Owen O'Malley added a comment - As we discussed offline, this is just caused because you didn't define a value for "hadoop.log.dir" on your java command line. It would be good to have the server not crash when the property is not defined, but the property should be defined for the servers. The two obvious "fixes" are to: 1. Remove the "/logs" context when no "hadoop.log.dir" property is defined. This will be confusing to users. 2. See if we can get the value from log4j via the jakarta commons logging api. It looks like there might be routines that will work. In any case, I believe this is a pretty low priority bug.
        Hide
        Devaraj Das added a comment -

        These were the exact same problems I encountered when I ran the unit tests. I had to define hadoop.log.dir (in the build.xml file) and had to put the "build" directory (which contains the webapps) in the CLASSPATH.

        Show
        Devaraj Das added a comment - These were the exact same problems I encountered when I ran the unit tests. I had to define hadoop.log.dir (in the build.xml file) and had to put the "build" directory (which contains the webapps) in the CLASSPATH.
        Hide
        navychen added a comment -

        here is my solution, does it okay for this bug?

        > 1. Remove the "/logs" context when no "hadoop.log.dir" property is defined.

        I changed StatusHttpServer.java at line 73-79
        73 // set up the context for "/logs/"
        74 HttpContext logContext = new HttpContext();
        75 logContext.setContextPath("/logs/*");
        76 String logDir = System.getProperty("hadoop.log.dir");
        77 logContext.setResourceBase(logDir);
        78 logContext.addHandler(new ResourceHandler());
        79 webServer.addContext(logContext);
        to
        73 // set up the context for "/logs/" if "hadoop.log.dir" property is defined.
        74 String logDir = System.getProperty("hadoop.log.dir");
        75 if( logDir != null )

        { 76 HttpContext logContext = new HttpContext(); 77 logContext.setContextPath("/logs/*"); 78 logContext.setResourceBase(logDir); 79 logContext.addHandler(new ResourceHandler()); 80 webServer.addContext(logContext); 81 }

        And it works okay, but there has a little impact that user cannot visit "/logs/" at web UI.

        > 2. See if we can get the value from log4j via the jakarta commons logging api. It looks like there might be routines that will work.

        I does not find a api to get these value at Jakarta commons logging api documents:
        http://jakarta.apache.org/commons/logging/apidocs/index.html
        I am puzzle that how to do this, because log4j.properties is read by log4j lib automatically,
        For hadoop framework, it don't know or should don't know what log library it use, so is it not good to
        Get value from log4j?

        > Konstantin:
        1, Even if I get through this I get yet another NullPointerException.Now because webapps resource is not defined.

        This is because he did not add base directory of hadoop that contains "webapp" directory into CLASSPATH,
        I insert two lines to StatusHttpServer.java to throw "not found webapps" exception to user, Is this a good way to fix it?
        154 private static String getWebAppsPath() throws IOException {
        155 URL url = StatusHttpServer.class.getClassLoader().getResource("webapps");
        156 if( url == null )
        157 throw new IOException("webapps not found in CLASSPATH");
        158 String path = url.getPath();

        if webapps resource is not defined, It seems that could not set a default value for it or return null,
        because webAppContext member of StatusHttpServer should be initialized with webapps resource path for many functions need to access it,
        so maybe throw a intelligible exception to user is a better way.

        > 2, trying to start the Namenode directly without using any scripts.

        To resolve this problem and let user can start NameNode without any scripts,
        can run as these steps in hadoop base directory:
        1, export CLASSPATH=$CLASSPATH:./build/:./build/classes/:./:./conf/
        2, export CLASSPATH=$CLASSPATH:`ls lib/*.jar | cat | tr '\n' :`
        3, export CLASSPATH=$CLASSPATH:`ls lib/jetty-ext/*.jar | cat | tr '\n' :`
        4, java org.apache.hadoop.dfs.NameNode

        Show
        navychen added a comment - here is my solution, does it okay for this bug? > 1. Remove the "/logs" context when no "hadoop.log.dir" property is defined. I changed StatusHttpServer.java at line 73-79 73 // set up the context for "/logs/" 74 HttpContext logContext = new HttpContext(); 75 logContext.setContextPath("/logs/*"); 76 String logDir = System.getProperty("hadoop.log.dir"); 77 logContext.setResourceBase(logDir); 78 logContext.addHandler(new ResourceHandler()); 79 webServer.addContext(logContext); to 73 // set up the context for "/logs/" if "hadoop.log.dir" property is defined. 74 String logDir = System.getProperty("hadoop.log.dir"); 75 if( logDir != null ) { 76 HttpContext logContext = new HttpContext(); 77 logContext.setContextPath("/logs/*"); 78 logContext.setResourceBase(logDir); 79 logContext.addHandler(new ResourceHandler()); 80 webServer.addContext(logContext); 81 } And it works okay, but there has a little impact that user cannot visit "/logs/" at web UI. > 2. See if we can get the value from log4j via the jakarta commons logging api. It looks like there might be routines that will work. I does not find a api to get these value at Jakarta commons logging api documents: http://jakarta.apache.org/commons/logging/apidocs/index.html I am puzzle that how to do this, because log4j.properties is read by log4j lib automatically, For hadoop framework, it don't know or should don't know what log library it use, so is it not good to Get value from log4j? > Konstantin: 1, Even if I get through this I get yet another NullPointerException.Now because webapps resource is not defined. This is because he did not add base directory of hadoop that contains "webapp" directory into CLASSPATH, I insert two lines to StatusHttpServer.java to throw "not found webapps" exception to user, Is this a good way to fix it? 154 private static String getWebAppsPath() throws IOException { 155 URL url = StatusHttpServer.class.getClassLoader().getResource("webapps"); 156 if( url == null ) 157 throw new IOException("webapps not found in CLASSPATH"); 158 String path = url.getPath(); if webapps resource is not defined, It seems that could not set a default value for it or return null, because webAppContext member of StatusHttpServer should be initialized with webapps resource path for many functions need to access it, so maybe throw a intelligible exception to user is a better way. > 2, trying to start the Namenode directly without using any scripts. To resolve this problem and let user can start NameNode without any scripts, can run as these steps in hadoop base directory: 1, export CLASSPATH=$CLASSPATH:./build/:./build/classes/:./:./conf/ 2, export CLASSPATH=$CLASSPATH:`ls lib/*.jar | cat | tr '\n' :` 3, export CLASSPATH=$CLASSPATH:`ls lib/jetty-ext/*.jar | cat | tr '\n' :` 4, java org.apache.hadoop.dfs.NameNode
        Hide
        navychen added a comment -

        this patch will fix the follow bugs:
        1, The NullPointerException happens in the constructor of StatusHttpServer because the System property "hadoop.log.dir" is undefined.

        • to fix this, remove the "/logs" context when no "hadoop.log.dir" property is defined.
          2, Even if I get through this I get yet another NullPointerException. Now because webapps resource is not defined.
        • to fix this, just throw a "not found webapps" IOException to client
        Show
        navychen added a comment - this patch will fix the follow bugs: 1, The NullPointerException happens in the constructor of StatusHttpServer because the System property "hadoop.log.dir" is undefined. to fix this, remove the "/logs" context when no "hadoop.log.dir" property is defined. 2, Even if I get through this I get yet another NullPointerException. Now because webapps resource is not defined. to fix this, just throw a "not found webapps" IOException to client
        Hide
        Doug Cutting added a comment -

        I just committed this. Thanks, navychen!

        Show
        Doug Cutting added a comment - I just committed this. Thanks, navychen!

          People

          • Assignee:
            navychen
            Reporter:
            Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development