Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1082

Command line UI for queues' information is broken with hierarchical queues.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Duplicate
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: client, jobtracker
    • Labels:
      None

      Description

      When the command "./bin/mapred --config ~/tmp/conf/ queue -list" is run, it just hangs. I can see the following in the JT logs:

      2009-10-08 13:19:26,762 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 55555 caught: java.lang.NullPointerException
              at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:217)
              at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:223)
              at org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:159)
              at org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:126)
              at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:70)
              at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1074)
              at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:983)
      

      Same is the case with "./bin/mapred --config ~/tmp/conf/ queue -info <any-container-queue>"

      1. MAPREDUCE-1082-1.txt
        6 kB
        V.V.Chaitanya Krishna
      2. MAPREDUCE-1082-2.patch
        6 kB
        V.V.Chaitanya Krishna
      3. MAPREDUCE-1082-3.patch
        5 kB
        V.V.Chaitanya Krishna
      4. MAPREDUCE-1082-4.patch
        6 kB
        V.V.Chaitanya Krishna

        Issue Links

          Activity

          Vinod Kumar Vavilapalli created issue -
          V.V.Chaitanya Krishna made changes -
          Field Original Value New Value
          Link This issue incorporates MAPREDUCE-1075 [ MAPREDUCE-1075 ]
          V.V.Chaitanya Krishna made changes -
          Assignee V.V.Chaitanya Krishna [ chaitk ]
          V.V.Chaitanya Krishna made changes -
          Link This issue incorporates MAPREDUCE-1075 [ MAPREDUCE-1075 ]
          V.V.Chaitanya Krishna made changes -
          Link This issue is blocked by MAPREDUCE-1007 [ MAPREDUCE-1007 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch. This issue requires patch for MAPREDUCE-1007, which is included in this patch for review.

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch. This issue requires patch for MAPREDUCE-1007 , which is included in this patch for review.
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1082-1.txt [ 12421964 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I've looked at the attached patch. It has some problems. The most important one is that for converting QueueInfo obects to JobQueueInfo objects, this patch creates a new JobQueueInfo objects everytime. This causes other tests like TestQueueMangerRefresh to fail. This has also proved to be problematic in the past too while working with MAPREDUCE-893.

          Going back and summarizing the main cause for this bug, the NPE occurs while trying to serialize a hierarchy of JobQueueInfo objects. JobQueueInfo extends QueueInfo and so serialization code is accordingly inherited. So, the problem is as follows:

          Hierarchy before serialization:

          JobQueueInfo-one
             |-- JobQueueInfo-child-one
             `-- JobQueueInfo-child-two
          

          Deserialized hierarchy:

          JobQueueInfo-one
             |-- QueueInfo-child-one
             `-- QueueInfo-child-two
          

          This poses problem while traversing the deserializing hierarchy.

          Two solutions possible:

          • From the deserialized hierarchy, construct a proper hierarchy of {{JobQueueInfo}}s and use this from then on.
          • Or solve the central probem which is about fixing the inconsistent handling of relation JobQueueInfo with QueueInfo objects.

          Given that the central problem has already resulted in some issues before too, I propose that once and for all we fix the problems with JobQueueInfo <-> QueueInfo inter-conversion as part of this JIRA issue. Thoughts?

          Show
          Vinod Kumar Vavilapalli added a comment - I've looked at the attached patch. It has some problems. The most important one is that for converting QueueInfo obects to JobQueueInfo objects, this patch creates a new JobQueueInfo objects everytime. This causes other tests like TestQueueMangerRefresh to fail. This has also proved to be problematic in the past too while working with MAPREDUCE-893 . Going back and summarizing the main cause for this bug, the NPE occurs while trying to serialize a hierarchy of JobQueueInfo objects. JobQueueInfo extends QueueInfo and so serialization code is accordingly inherited. So, the problem is as follows: Hierarchy before serialization: JobQueueInfo-one |-- JobQueueInfo-child-one `-- JobQueueInfo-child-two Deserialized hierarchy: JobQueueInfo-one |-- QueueInfo-child-one `-- QueueInfo-child-two This poses problem while traversing the deserializing hierarchy. Two solutions possible: From the deserialized hierarchy, construct a proper hierarchy of {{JobQueueInfo}}s and use this from then on. Or solve the central probem which is about fixing the inconsistent handling of relation JobQueueInfo with QueueInfo objects. Given that the central problem has already resulted in some issues before too, I propose that once and for all we fix the problems with JobQueueInfo <-> QueueInfo inter-conversion as part of this JIRA issue. Thoughts?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Thinking more about it, I am fine with just fixing the NPE and the problem with JobQueueInfo.readFields() to solve this issue. We can delay the cleanup for later via MAPREDUCE-1110.

          Show
          Vinod Kumar Vavilapalli added a comment - Thinking more about it, I am fine with just fixing the NPE and the problem with JobQueueInfo.readFields() to solve this issue. We can delay the cleanup for later via MAPREDUCE-1110 .
          Hide
          V.V.Chaitanya Krishna added a comment -

          uploading patch with the QueueInfo.readFields() overriden in JobQueueInfo.
          The patch for MAPREDUCE-1007 to be applied before this patch is applied.

          Show
          V.V.Chaitanya Krishna added a comment - uploading patch with the QueueInfo.readFields() overriden in JobQueueInfo. The patch for MAPREDUCE-1007 to be applied before this patch is applied.
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1082-2.patch [ 12424575 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          This patch is changing the serialization format of the deprecated JobQueueInfo. Not sure if it is/isn't OK.

          In any case, back-compatibility of JobQueueInfo itself is broken after the incompatible changes done by MAPREDUCE-777; for e.g. some of the public methods are no longer available and have protected scope now.

          Show
          Vinod Kumar Vavilapalli added a comment - This patch is changing the serialization format of the deprecated JobQueueInfo . Not sure if it is/isn't OK. In any case, back-compatibility of JobQueueInfo itself is broken after the incompatible changes done by MAPREDUCE-777 ; for e.g. some of the public methods are no longer available and have protected scope now.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Or should I say MAPREDUCE-861 broke the back-compatibility of the serialization.

          Show
          Vinod Kumar Vavilapalli added a comment - Or should I say MAPREDUCE-861 broke the back-compatibility of the serialization.
          Vinod Kumar Vavilapalli made changes -
          Link This issue blocks MAPREDUCE-1110 [ MAPREDUCE-1110 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I am thinking of addressing the backward-compatibility story at MAPREDUCE-1110.
          Given that, I am +1 for this patch.

          Submitting it to Hudson..

          Show
          Vinod Kumar Vavilapalli added a comment - I am thinking of addressing the backward-compatibility story at MAPREDUCE-1110 . Given that, I am +1 for this patch. Submitting it to Hudson..
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424575/MAPREDUCE-1082-2.patch
          against trunk revision 882470.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424575/MAPREDUCE-1082-2.patch against trunk revision 882470. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/console This message is automatically generated.
          Hide
          Hemanth Yamijala added a comment -

          Have a few comments on this patch:

          • I don't know why readFields in JobQueueInfo needs to be overridden. In all the API, it is QueueInfo objects that are transferred over IPC, and I think it should remain that way.
          • I think the test case needs to be an end-to-end test by definition, as the fix is both in JobQueueInfo as well as in the JobTracker, where when we translate JobQueueInfos to QueueInfos, we fix the translation by walking the entire hierarchy. I would suggest a test that brings up a MiniMRCluster with hierarchical queues, submits a job to one of the queues and calls Cluster.getRootQueues and verifies the returned QueueInfo information. We might need a package private JobTracker.setQueueManager to enable setting up hierarchical queues with a miniMRCluster.
          Show
          Hemanth Yamijala added a comment - Have a few comments on this patch: I don't know why readFields in JobQueueInfo needs to be overridden. In all the API, it is QueueInfo objects that are transferred over IPC, and I think it should remain that way. I think the test case needs to be an end-to-end test by definition, as the fix is both in JobQueueInfo as well as in the JobTracker, where when we translate JobQueueInfos to QueueInfos, we fix the translation by walking the entire hierarchy. I would suggest a test that brings up a MiniMRCluster with hierarchical queues, submits a job to one of the queues and calls Cluster.getRootQueues and verifies the returned QueueInfo information. We might need a package private JobTracker.setQueueManager to enable setting up hierarchical queues with a miniMRCluster.
          V.V.Chaitanya Krishna made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch with the above ideas implemented.

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch with the above ideas implemented.
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1082-3.patch [ 12426514 ]
          Hide
          Hemanth Yamijala added a comment -

          Looking close. Some final comments:

          • We are assuming the job statuses cannot be null in QueueInfo. I think we should check this in setJobStatuses. If it is null, we can set an empty array.
          • The test case should call APIs like setRootQueues. getQueue is not passing through the code path change you made in JobTracker.getQueueInfoArray
          Show
          Hemanth Yamijala added a comment - Looking close. Some final comments: We are assuming the job statuses cannot be null in QueueInfo. I think we should check this in setJobStatuses. If it is null, we can set an empty array. The test case should call APIs like setRootQueues. getQueue is not passing through the code path change you made in JobTracker.getQueueInfoArray
          Hide
          V.V.Chaitanya Krishna added a comment -

          While modifying the test case to implement the above comments, it was found that QueueInfo.readFields() recursively creates child objects as QueueInfo instances. This would lead to error while casting to JobQueueInfo in JobQueueInfo.getChildren().

          Uploading patch with this handled.

          Show
          V.V.Chaitanya Krishna added a comment - While modifying the test case to implement the above comments, it was found that QueueInfo.readFields() recursively creates child objects as QueueInfo instances. This would lead to error while casting to JobQueueInfo in JobQueueInfo.getChildren(). Uploading patch with this handled.
          V.V.Chaitanya Krishna made changes -
          Attachment MAPREDUCE-1082-4.patch [ 12427144 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Had an offline discussion with Hemanth and Vinod.
          The issues that are being found here can all be solved with the fix to MAPREDUCE-1110. So, resolving this issue as duplicate of it.

          Show
          V.V.Chaitanya Krishna added a comment - Had an offline discussion with Hemanth and Vinod. The issues that are being found here can all be solved with the fix to MAPREDUCE-1110 . So, resolving this issue as duplicate of it.
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is part of MAPREDUCE-1110 [ MAPREDUCE-1110 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          43d 2h 22m 1 Vinod Kumar Vavilapalli 20/Nov/09 10:18
          Patch Available Patch Available Open Open
          10d 21h 29m 1 V.V.Chaitanya Krishna 01/Dec/09 07:48
          Open Open Resolved Resolved
          6d 21h 56m 1 V.V.Chaitanya Krishna 08/Dec/09 05:44
          Resolved Resolved Closed Closed
          259d 15h 33m 1 Tom White 24/Aug/10 22:18

            People

            • Assignee:
              V.V.Chaitanya Krishna
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development