Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3674

If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When you access /jobqueue_details.jsp manually, instead of via a link, it has queueName set to null internally and this goes for a lookup into the scheduling info maps as well.

      As a result, if using FairScheduler, a Pool with String name = null gets created and this brings the scheduler down. I have not tested what happens to the CapacityScheduler, but ideally if no queueName is set in that jsp, it should fall back to 'default'. Otherwise, this brings down the JobTracker completely.

      FairScheduler must also add a check to not create a pool with 'null' name.

      The following is the strace that ensues:

      ERROR org.mortbay.log: /jobqueue_details.jsp 
      java.lang.NullPointerException 
      at org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71) 
      at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) 
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) 
      at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) 
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) 
      at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829) 
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) 
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) 
      at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) 
      at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
      at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 
      at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) 
      at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) 
      at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
      at org.mortbay.jetty.Server.handle(Server.java:326) 
      at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) 
      at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) 
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) 
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) 
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) 
      at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) 
      at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 
      INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call heartbeat from XYZ:MNOP: error: java.io.IOException: java.lang.NullPointerException 
      java.io.IOException: java.lang.NullPointerException 
      at org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95) 
      at org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68) 
      at java.util.Arrays.mergeSort(Unknown Source) 
      at java.util.Arrays.sort(Unknown Source) 
      at java.util.Collections.sort(Unknown Source) 
      at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435) 
      at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226) 
      at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) 
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 
      at java.lang.reflect.Method.invoke(Unknown Source) 
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) 
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) 
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) 
      at java.security.AccessController.doPrivileged(Native Method) 
      at javax.security.auth.Subject.doAs(Unknown Source) 
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) 
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
      
      1. MAPREDUCE-3674.patch
        0.6 kB
        Gera Shegalov
      2. MAPREDUCE-3674.patch
        3 kB
        Harsh J
      3. hadoop-findbugs-report.html
        312 kB
        Harsh J
      4. MAPREDUCE-3674.patch
        3 kB
        Harsh J
      5. MAPREDUCE-3674.patch
        2 kB
        Harsh J
      6. MAPREDUCE-3674.patch
        2 kB
        Harsh J

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          Or rather, since problem appears to be in FairScheduler, I'll write a patch accordingly for that.

          I do not see CapacityScheduler doing dynamic queue names so it is safe.

          Show
          Harsh J added a comment - Or rather, since problem appears to be in FairScheduler, I'll write a patch accordingly for that. I do not see CapacityScheduler doing dynamic queue names so it is safe.
          Hide
          Harsh J added a comment -

          Manually tested patch to verify it does not inject null queuename pools into FairScheduler.

          Show
          Harsh J added a comment - Manually tested patch to verify it does not inject null queuename pools into FairScheduler.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12510664/MAPREDUCE-3674.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1614//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510664/MAPREDUCE-3674.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1614//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          hm, rather than replacing null with "", maybe better to just disallow it? That way we're more likely to notice if we have a bug later. Preconditions.checkNotNull or Preconditions.checkArgument perhaps.

          Show
          Todd Lipcon added a comment - hm, rather than replacing null with "", maybe better to just disallow it? That way we're more likely to notice if we have a bug later. Preconditions.checkNotNull or Preconditions.checkArgument perhaps.
          Hide
          Harsh J added a comment -

          Fixed. No Guava on 1.x branch so could not use the Precondition util.

          Show
          Harsh J added a comment - Fixed. No Guava on 1.x branch so could not use the Precondition util.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12511088/MAPREDUCE-3674.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1630//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511088/MAPREDUCE-3674.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1630//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12511088/MAPREDUCE-3674.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1655//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511088/MAPREDUCE-3674.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1655//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Ping, any further comments?

          Show
          Harsh J added a comment - Ping, any further comments?
          Hide
          Harsh J added a comment -

          New patch with one test added.

          -1 overall.  
          
              +1 @author.  The patch does not contain any @author tags.
          
              +1 tests included.  The patch appears to include 3 new or modified tests.
          
              +1 javadoc.  The javadoc tool did not generate any warning messages.
          
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          
              -1 findbugs.  The patch appears to introduce 196 new Findbugs (version 2.0.0) warnings.
          

          None of the 196 findbugs are me. Report attached.

          Show
          Harsh J added a comment - New patch with one test added. -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 196 new Findbugs (version 2.0.0) warnings. None of the 196 findbugs are me. Report attached.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521466/MAPREDUCE-3674.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2154//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521466/MAPREDUCE-3674.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2154//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521468/MAPREDUCE-3674.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2155//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521468/MAPREDUCE-3674.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2155//console This message is automatically generated.
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Harsh J added a comment -

          Thanks for the review Tom and Todd, committed for 1.1.0.

          Show
          Harsh J added a comment - Thanks for the review Tom and Todd, committed for 1.1.0.
          Hide
          Gera Shegalov added a comment -

          Fix to src/webapps/job/jobqueue_details.jsp
          to avoid NPE when invalid queue name, such as null or non-existing queue name is specified. It avoids null schedInfo.

          Show
          Gera Shegalov added a comment - Fix to src/webapps/job/jobqueue_details.jsp to avoid NPE when invalid queue name, such as null or non-existing queue name is specified. It avoids null schedInfo.
          Hide
          Harsh J added a comment -

          Hello Gera,

          Could you create a new issue for your patch and what it aims to fix (and what was missed in this original one here)?

          This JIRA has been resolved, so please open a new one with your patch and link back to this one as a relation.

          Thanks!

          Show
          Harsh J added a comment - Hello Gera, Could you create a new issue for your patch and what it aims to fix (and what was missed in this original one here)? This JIRA has been resolved, so please open a new one with your patch and link back to this one as a relation. Thanks!
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop-1.1.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop-1.1.0.

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development