Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1007

MAPREDUCE-777 breaks the UI for hierarchial Queues.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MAPREDUCE-777 breaks jobtracker UI for hierarchial queues. When jobtracker.jsp is accessed, it throws the following exception:

      java.lang.NullPointerException
      	at org.apache.hadoop.mapred.CapacityTaskScheduler.getJobs(CapacityTaskScheduler.java:1007)
      	at org.apache.hadoop.mapred.JobTracker.getJobsFromQueue(JobTracker.java:3888)
      	at org.apache.hadoop.mapred.JobTracker.getQueueInfoArray(JobTracker.java:3869)
      	at org.apache.hadoop.mapred.JobTracker.getRootQueues(JobTracker.java:3830)
      	at org.apache.hadoop.mapred.jobtracker_jsp.generateSummaryTable(jobtracker_jsp.java:36)
              .... 
      

      (Issue number and the line number in code match - 1007. Some fun for a Hadoop developer )

      1. MAPREDUCE-1007-8.txt
        8 kB
        V.V.Chaitanya Krishna
      2. MAPREDUCE-1007-7.txt
        8 kB
        V.V.Chaitanya Krishna
      3. MAPREDUCE-1007-6.txt
        8 kB
        V.V.Chaitanya Krishna
      4. MAPREDUCE-1007-5.txt
        8 kB
        V.V.Chaitanya Krishna
      5. MAPREDUCE-1007-4.txt
        8 kB
        V.V.Chaitanya Krishna
      6. MAPREDUCE-1007-3.txt
        7 kB
        V.V.Chaitanya Krishna
      7. MAPREDUCE-1007-2.txt
        8 kB
        V.V.Chaitanya Krishna
      8. MAPREDUCE-1007-1.patch
        3 kB
        V.V.Chaitanya Krishna
      9. MAPREDUCE-1007.patch
        0.7 kB
        rahul k singh

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          54d 19m 4 Vinod Kumar Vavilapalli 19/Nov/09 05:45
          Open Open Patch Available Patch Available
          7d 9h 26m 5 Vinod Kumar Vavilapalli 19/Nov/09 05:45
          Patch Available Patch Available Resolved Resolved
          1d 4h 15m 1 Sharad Agarwal 20/Nov/09 10:00
          Resolved Resolved Closed Closed
          277d 11h 17m 1 Tom White 24/Aug/10 22:17
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #149 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/149/)
          . Fix NPE in CapacityTaskScheduler.getJobs(). Contributed by V.V.Chaitanya Krishna.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #149 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/149/ ) . Fix NPE in CapacityTaskScheduler.getJobs(). Contributed by V.V.Chaitanya Krishna.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #124 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/124/)
          . Fix NPE in CapacityTaskScheduler.getJobs(). Contributed by V.V.Chaitanya Krishna.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #124 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/124/ ) . Fix NPE in CapacityTaskScheduler.getJobs(). Contributed by V.V.Chaitanya Krishna.
          Sharad Agarwal made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Sharad Agarwal added a comment -

          I just committed this. Thanks Chaitanya!

          Show
          Sharad Agarwal added a comment - I just committed this. Thanks Chaitanya!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12422061/MAPREDUCE-1007-8.txt
          against trunk revision 881673.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/252/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/252/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/252/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/252/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422061/MAPREDUCE-1007-8.txt against trunk revision 881673. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/252/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/252/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/252/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/252/console This message is automatically generated.
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I'm asking Arun to commit this. Will run it through Hudson again to be sure.

          Show
          Vinod Kumar Vavilapalli added a comment - I'm asking Arun to commit this. Will run it through Hudson again to be sure.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12422061/MAPREDUCE-1007-8.txt
          against trunk revision 825083.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/167/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/167/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/167/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/167/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422061/MAPREDUCE-1007-8.txt against trunk revision 825083. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/167/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/167/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/167/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/167/console This message is automatically generated.
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          V.V.Chaitanya Krishna made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1007-8.txt [ 12422061 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Since MAPREDUCE-1030 has been comitted, the previous patch wont apply. Uploading new patch.

          Show
          V.V.Chaitanya Krishna added a comment - Since MAPREDUCE-1030 has been comitted, the previous patch wont apply. Uploading new patch.
          V.V.Chaitanya Krishna made changes -
          Link This issue blocks MAPREDUCE-1082 [ MAPREDUCE-1082 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12421719/MAPREDUCE-1007-7.txt
          against trunk revision 823227.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/68/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/68/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/68/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/68/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12421719/MAPREDUCE-1007-7.txt against trunk revision 823227. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/68/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/68/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/68/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/68/console This message is automatically generated.
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          rahul k singh added a comment -

          +1

          Show
          rahul k singh added a comment - +1
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1007-7.txt [ 12421719 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          The check for queueManager being null is redundant in the previous patch. Uploading patch with this rectified.

          Show
          V.V.Chaitanya Krishna added a comment - The check for queueManager being null is redundant in the previous patch. Uploading patch with this rectified.
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1007-6.txt [ 12421619 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          The previous patch has redundant check for queueManager.getJobQueueInfo(queue) being null.
          Uploading patch with this removed

          Show
          V.V.Chaitanya Krishna added a comment - The previous patch has redundant check for queueManager.getJobQueueInfo(queue) being null. Uploading patch with this removed
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1007-5.txt [ 12421615 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch with corrections related to the above comment.

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch with corrections related to the above comment.
          V.V.Chaitanya Krishna made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          rahul k singh added a comment -

          patch seems fine , except 1 minor nit.

          1. in getJobsFromQueue if unvalid queueName is passed it would throw NPE.
          instead we should check for this condition and throw exception.

          Show
          rahul k singh added a comment - patch seems fine , except 1 minor nit. 1. in getJobsFromQueue if unvalid queueName is passed it would throw NPE. instead we should check for this condition and throw exception.
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1007-4.txt [ 12421518 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch with the check for presence of children in getJobsFromQueue(String queue) method.
          The other comment will be handled in a separate jira (MAPREDUCE-1075).

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch with the check for presence of children in getJobsFromQueue(String queue) method. The other comment will be handled in a separate jira ( MAPREDUCE-1075 ).
          Hide
          rahul k singh added a comment -

          Some minor comments:

          1. Check if a queue has children or not in getJobsFromQueue(String queue) method.
          2. getQueue(String queue) in JobTracker would return NPE for invalid queue name(Not related to jira but still a bug).

          Show
          rahul k singh added a comment - Some minor comments: 1. Check if a queue has children or not in getJobsFromQueue(String queue) method. 2. getQueue(String queue) in JobTracker would return NPE for invalid queue name(Not related to jira but still a bug).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12420527/MAPREDUCE-1007-3.txt
          against trunk revision 818674.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/132/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/132/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/132/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/132/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420527/MAPREDUCE-1007-3.txt against trunk revision 818674. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/132/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/132/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/132/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/132/console This message is automatically generated.
          Hide
          V.V.Chaitanya Krishna added a comment -

          The -1 in core tests is due to the known issue of TestCopyFiles failing (ref: MAPREDUCE-1029).

          Show
          V.V.Chaitanya Krishna added a comment - The -1 in core tests is due to the known issue of TestCopyFiles failing (ref: MAPREDUCE-1029 ).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12420527/MAPREDUCE-1007-3.txt
          against trunk revision 818674.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/131/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/131/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/131/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/131/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420527/MAPREDUCE-1007-3.txt against trunk revision 818674. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/131/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/131/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/131/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/131/console This message is automatically generated.
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Patch looks good. +1. Running the latest patch through Hudson.

          Show
          Vinod Kumar Vavilapalli added a comment - Patch looks good. +1. Running the latest patch through Hudson.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12420524/MAPREDUCE-1007-2.txt
          against trunk revision 818674.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/130/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/130/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/130/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/130/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420524/MAPREDUCE-1007-2.txt against trunk revision 818674. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/130/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/130/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/130/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/130/console This message is automatically generated.
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1007-3.txt [ 12420527 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Had an offline discussion with Vinod with few comments. Uploading new patch with these comments implemented.

          Show
          V.V.Chaitanya Krishna added a comment - Had an offline discussion with Vinod with few comments. Uploading new patch with these comments implemented.
          V.V.Chaitanya Krishna made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1007-2.txt [ 12420524 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch with above mentioned comments considered.

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch with above mentioned comments considered.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Quickly looked at the patch and tested it on a single node. It works fine. Some review comments, mostly minor:

          • In CapacityTaskScheduler.getJobs(), we can cache the return value from jobQueuesManager.getJobQueue(queueName) to avoid repetitive lookups.
          • Test-cases testSubmitToQueues(), testGetJobs() and the newly added testJobsForContainerQueues() share a lot of common stuff and test a single concept - job submission to capacity-scheduler. They can be combined into a single testJobSubmission(). Further, instead of creating all the internal queue-related data-structures ourselves, we can simply create a configuration file and start the scheduler. See TestRefreshOfQueues.testSuccessfulCapacityRefresh() for an example. You may need to do some refactoring to facilitate this.

          One orthogonal point which this issue may not concern itself with, jobtracker.jsp is now printing the number of root-queues under a header 'queues'. Just looking at it, it didn't tell me what it actually represents. We can (1) rename it to root-queues to be clear, or (2) print both root-queues' and job-queues' number or (3) do away with these numbers altogether and just give a hyper-link to the queues page. Thoughts?

          Show
          Vinod Kumar Vavilapalli added a comment - Quickly looked at the patch and tested it on a single node. It works fine. Some review comments, mostly minor: In CapacityTaskScheduler.getJobs(), we can cache the return value from jobQueuesManager.getJobQueue(queueName) to avoid repetitive lookups. Test-cases testSubmitToQueues(), testGetJobs() and the newly added testJobsForContainerQueues() share a lot of common stuff and test a single concept - job submission to capacity-scheduler. They can be combined into a single testJobSubmission(). Further, instead of creating all the internal queue-related data-structures ourselves, we can simply create a configuration file and start the scheduler. See TestRefreshOfQueues.testSuccessfulCapacityRefresh() for an example. You may need to do some refactoring to facilitate this. One orthogonal point which this issue may not concern itself with, jobtracker.jsp is now printing the number of root-queues under a header 'queues'. Just looking at it, it didn't tell me what it actually represents. We can (1) rename it to root-queues to be clear, or (2) print both root-queues' and job-queues' number or (3) do away with these numbers altogether and just give a hyper-link to the queues page. Thoughts?
          Hide
          V.V.Chaitanya Krishna added a comment -

          The UI is not being displayed when a hierarchy of queues in built with atleast one container queue (i.e., atleast one non-leaf queue).
          There is no check for occurrence of null pointer in CapacityTaskScheduler.getJobs(queueName) and when the container queue's name is given as input parameter, it fails with NPE.

          The above patch is to handle this issue. It also includes test cases written to check the behaviour when a job is submitted to a container queue.

          Show
          V.V.Chaitanya Krishna added a comment - The UI is not being displayed when a hierarchy of queues in built with atleast one container queue (i.e., atleast one non-leaf queue). There is no check for occurrence of null pointer in CapacityTaskScheduler.getJobs(queueName) and when the container queue's name is given as input parameter, it fails with NPE. The above patch is to handle this issue. It also includes test cases written to check the behaviour when a job is submitted to a container queue.
          Vinod Kumar Vavilapalli made changes -
          Summary MAPREDUCE 777 breaks the UI for hierarchial Queues. MAPREDUCE-777 breaks the UI for hierarchial Queues.
          Description mapreduce 777 breaks jobtracker UI for hierarchial queues
          MAPREDUCE-777 breaks jobtracker UI for hierarchial queues. When jobtracker.jsp is accessed, it throws the following exception:

          {code}
          java.lang.NullPointerException
          at org.apache.hadoop.mapred.CapacityTaskScheduler.getJobs(CapacityTaskScheduler.java:1007)
          at org.apache.hadoop.mapred.JobTracker.getJobsFromQueue(JobTracker.java:3888)
          at org.apache.hadoop.mapred.JobTracker.getQueueInfoArray(JobTracker.java:3869)
          at org.apache.hadoop.mapred.JobTracker.getRootQueues(JobTracker.java:3830)
          at org.apache.hadoop.mapred.jobtracker_jsp.generateSummaryTable(jobtracker_jsp.java:36)
                  ....
          {code}
          (Issue number and the line number in code match - 1007. Some fun for a Hadoop developer :) )
          Vinod Kumar Vavilapalli made changes -
          Assignee V.V.Chaitanya Krishna [ chaitk ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Affects Version/s 0.21.0 [ 12314045 ]
          Component/s jobtracker [ 12312907 ]
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1007-1.patch [ 12420440 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch with test case.

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch with test case.
          rahul k singh made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-1007.patch [ 12420072 ]
          Hide
          rahul k singh added a comment -

          attaching a preliminary patch , needs more testing, it needs to be applied on 893

          Show
          rahul k singh added a comment - attaching a preliminary patch , needs more testing, it needs to be applied on 893
          rahul k singh created issue -

            People

            • Assignee:
              V.V.Chaitanya Krishna
              Reporter:
              rahul k singh
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development