Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      - The command line of hadoop queue -list and -info was changed to support hierarchical queues. So, they would now print information about child queues, wherever relevant.
      - The Web UI of the JobTracker was changed to list queues and queue information in a separate page.
      Show
      - The command line of hadoop queue -list and -info was changed to support hierarchical queues. So, they would now print information about child queues, wherever relevant. - The Web UI of the JobTracker was changed to list queues and queue information in a separate page.

      Description

      MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce framework. This JIRA is for defining changes to the UI related to queues. This includes the hadoop queue CLI and the web UI on the JobTracker.

      1. clustersummarymodification.png
        23 kB
        Sreekanth Ramakrishnan
      2. detailspage.png
        152 kB
        Sreekanth Ramakrishnan
      3. initialscreen.png
        83 kB
        Sreekanth Ramakrishnan
      4. subqueue.png
        107 kB
        Sreekanth Ramakrishnan
      5. MAPREDUCE-862-1.patch
        187 kB
        V.V.Chaitanya Krishna
      6. MAPREDUCE-862-2.patch
        7 kB
        V.V.Chaitanya Krishna
      7. MAPREDUCE-862-3.patch
        17 kB
        V.V.Chaitanya Krishna
      8. MAPREDUCE-862-4.patch
        33 kB
        V.V.Chaitanya Krishna
      9. MAPREDUCE-862-5.patch
        37 kB
        V.V.Chaitanya Krishna
      10. MAPREDUCE-862-6.patch
        32 kB
        V.V.Chaitanya Krishna
      11. MAPREDUCE-862-7.patch
        30 kB
        V.V.Chaitanya Krishna

        Activity

        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching screens of how the UI would look for modified queue design.

        Cluster summary would be modified, to introduce a new column which will have a number of Queue, which will be linked to modified queue details page, which is described in initialscreen.png.

        From initalscreen.png we can click on the queue hierarchy, which would have two pages, for ContainerQueues we would not have a job list and for JobQueue we have a job list apart from scheduling information.

        Show
        Sreekanth Ramakrishnan added a comment - Attaching screens of how the UI would look for modified queue design. Cluster summary would be modified, to introduce a new column which will have a number of Queue, which will be linked to modified queue details page, which is described in initialscreen.png. From initalscreen.png we can click on the queue hierarchy, which would have two pages, for ContainerQueues we would not have a job list and for JobQueue we have a job list apart from scheduling information.
        Hide
        Hemanth Yamijala added a comment -

        The mock-ups look fine to me.

        For the benefit of others, these were created using YUI that is available under the BSD license which is an Apache compatible license.

        In a local discussion with folks, we decided that we could just let the YUI scripts be downloaded in a standard manner by the browser. This means that to view the queue related pages, we'd need a working internet connection. That seems a reasonable assumption to make, with a caveat that if there's no connection, it should fail gracefully. Other options to bundle the javascripts would be involve a lot of sync-up with YUI libraries, which was felt a hazzle to maintain.

        Thoughts ?

        Show
        Hemanth Yamijala added a comment - The mock-ups look fine to me. For the benefit of others, these were created using YUI that is available under the BSD license which is an Apache compatible license. In a local discussion with folks, we decided that we could just let the YUI scripts be downloaded in a standard manner by the browser. This means that to view the queue related pages, we'd need a working internet connection. That seems a reasonable assumption to make, with a caveat that if there's no connection, it should fail gracefully. Other options to bundle the javascripts would be involve a lot of sync-up with YUI libraries, which was felt a hazzle to maintain. Thoughts ?
        Hide
        V.V.Chaitanya Krishna added a comment -

        Uploading the patch with the UI design as proposed in the above comments.

        Show
        V.V.Chaitanya Krishna added a comment - Uploading the patch with the UI design as proposed in the above comments.
        Hide
        V.V.Chaitanya Krishna added a comment -

        The uploaded patch contains the patch of MAPREDUCE-861 since it is required for the design.

        Show
        V.V.Chaitanya Krishna added a comment - The uploaded patch contains the patch of MAPREDUCE-861 since it is required for the design.
        Hide
        Sreekanth Ramakrishnan added a comment -

        Took a look at patch:

        • Add link in cluster status table to redirecting to new queue information page.
        • Move the Queue information to a new JSP. redirect to this page if we are not able to connect to YUI servers.

        Hemanth's offline comment:

        • Add recursive option in CLI for queue -info <queue-name>
        • Add a new field in the queue -info <queue-name> which lists only top level child queues.
        Show
        Sreekanth Ramakrishnan added a comment - Took a look at patch: Add link in cluster status table to redirecting to new queue information page. Move the Queue information to a new JSP. redirect to this page if we are not able to connect to YUI servers. Hemanth's offline comment: Add recursive option in CLI for queue -info <queue-name> Add a new field in the queue -info <queue-name> which lists only top level child queues.
        Hide
        V.V.Chaitanya Krishna added a comment -

        Had an offline discussion with Hemanth and Sreekanth. The recursive option for info is not required.

        Uploading the patch with the other above comments implemented.

        Show
        V.V.Chaitanya Krishna added a comment - Had an offline discussion with Hemanth and Sreekanth. The recursive option for info is not required. Uploading the patch with the other above comments implemented.
        Hide
        Hemanth Yamijala added a comment -

        I looked at the CLI changes. Look mostly fine. Few minor comments:

        • displayQueueInfo: Redundant call to jobQueueInfo.getChildren can be removed.
        • Also, I think child queues should be printed before the jobs are shown as its more natural that way
        • Should we print the full name of the queue when displaying the queue name ? I think it will be convenient to do so.
        • Also, I think it will be useful to separate each queue info with a separator like "====="
        • I think we should try and write test cases using TestMRCli.
        Show
        Hemanth Yamijala added a comment - I looked at the CLI changes. Look mostly fine. Few minor comments: displayQueueInfo: Redundant call to jobQueueInfo.getChildren can be removed. Also, I think child queues should be printed before the jobs are shown as its more natural that way Should we print the full name of the queue when displaying the queue name ? I think it will be convenient to do so. Also, I think it will be useful to separate each queue info with a separator like "=====" I think we should try and write test cases using TestMRCli.
        Hide
        Sreekanth Ramakrishnan added a comment -

        The new files which were added for the UI are missing in the latest patch, can you reupload the patch with those files included?
        In jobtracker.jsp after removing of the scheduling information table, please add a <hr/> to terminate the section from Summary table.

        Show
        Sreekanth Ramakrishnan added a comment - The new files which were added for the UI are missing in the latest patch, can you reupload the patch with those files included? In jobtracker.jsp after removing of the scheduling information table, please add a <hr/> to terminate the section from Summary table.
        Hide
        V.V.Chaitanya Krishna added a comment -

        Uploading patch with the above comments of Hemanth and Sreekanth implemented except for the test cases.
        The test cases are being worked and will be put up in the next patch.

        Show
        V.V.Chaitanya Krishna added a comment - Uploading patch with the above comments of Hemanth and Sreekanth implemented except for the test cases. The test cases are being worked and will be put up in the next patch.
        Hide
        V.V.Chaitanya Krishna added a comment -

        Uploading the patch with the test case being implemented.

        Show
        V.V.Chaitanya Krishna added a comment - Uploading the patch with the test case being implemented.
        Hide
        V.V.Chaitanya Krishna added a comment -

        Had an offline discussion with Sreekanth regarding the UI. Some minor corrections were identified. Uploading the patch with these corrections done.

        Show
        V.V.Chaitanya Krishna added a comment - Had an offline discussion with Sreekanth regarding the UI. Some minor corrections were identified. Uploading the patch with these corrections done.
        Hide
        Hemanth Yamijala added a comment -

        Regarding the CLI changes, I think we must:

        • Minimize the changes to JobQueueClient and not make any changes to JobClient - because we aren't per se changing anything in the job client.
        • Convert the unit test to a proper unit test without needing a MiniMRCluster.
        Show
        Hemanth Yamijala added a comment - Regarding the CLI changes, I think we must: Minimize the changes to JobQueueClient and not make any changes to JobClient - because we aren't per se changing anything in the job client. Convert the unit test to a proper unit test without needing a MiniMRCluster.
        Hide
        V.V.Chaitanya Krishna added a comment -

        Uploading patch with the above points mentioned by Hemanth implemented.

        Show
        V.V.Chaitanya Krishna added a comment - Uploading patch with the above points mentioned by Hemanth implemented.
        Hide
        Hemanth Yamijala added a comment -

        This is quite close. I have a few cleanup comments:

        1. JobQueueClient.displayQueueAclsInfoForCurrentUser need not init JobConf again.
        2. Are the changes in QueueManagerTestUtils required ?
        3. CONFDIR and CONFFILEPATH in TestJobQueueClient seem unnecessary now.

        Please mark this patch available with the above changes.

        Show
        Hemanth Yamijala added a comment - This is quite close. I have a few cleanup comments: 1. JobQueueClient.displayQueueAclsInfoForCurrentUser need not init JobConf again. 2. Are the changes in QueueManagerTestUtils required ? 3. CONFDIR and CONFFILEPATH in TestJobQueueClient seem unnecessary now. Please mark this patch available with the above changes.
        Hide
        V.V.Chaitanya Krishna added a comment -

        Uploading patch with the above changes made and making it patch available.

        Show
        V.V.Chaitanya Krishna added a comment - Uploading patch with the above changes made and making it patch available.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12419845/MAPREDUCE-862-7.patch
        against trunk revision 816052.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419845/MAPREDUCE-862-7.patch against trunk revision 816052. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/94/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this. Thanks, Chaitanya !

        Show
        Hemanth Yamijala added a comment - I just committed this. Thanks, Chaitanya !
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #45 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/45/)
        . Enhance JobTracker UI to display hierarchical queues. Contributed by V.V.Chaitanya Krishna.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #45 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/45/ ) . Enhance JobTracker UI to display hierarchical queues. Contributed by V.V.Chaitanya Krishna.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/85/)
        . Enhance JobTracker UI to display hierarchical queues. Contributed by V.V.Chaitanya Krishna.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/85/ ) . Enhance JobTracker UI to display hierarchical queues. Contributed by V.V.Chaitanya Krishna.
        Hide
        Aaron Kimball added a comment -

        -1. My jobtracker now fails to start with a NullPointerException:

        2009-09-17 10:45:46,121 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
        2009-09-17 10:45:46,124 FATAL org.apache.hadoop.mapred.JobTracker: java.lang.NullPointerException
        »- at org.apache.hadoop.mapred.QueueManager.getQueueConfigurationParser(QueueManager.java:117)
        »- at org.apache.hadoop.mapred.QueueManager.<init>(QueueManager.java:137)
        »- at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1315)
        »- at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:212)
        »- at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:204)
        »- at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:199)
        »- at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3584)
        
        2009-09-17 10:45:46,125 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
        

        This happens if conf/mapred-queues.xml does not exist.

        Show
        Aaron Kimball added a comment - -1. My jobtracker now fails to start with a NullPointerException: 2009-09-17 10:45:46,121 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list 2009-09-17 10:45:46,124 FATAL org.apache.hadoop.mapred.JobTracker: java.lang.NullPointerException »- at org.apache.hadoop.mapred.QueueManager.getQueueConfigurationParser(QueueManager.java:117) »- at org.apache.hadoop.mapred.QueueManager.<init>(QueueManager.java:137) »- at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1315) »- at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:212) »- at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:204) »- at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:199) »- at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3584) 2009-09-17 10:45:46,125 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG: This happens if conf/mapred-queues.xml does not exist.
        Hide
        Hemanth Yamijala added a comment -

        Aaron, in Hadoop 0.21, we are shifting the queue configuration to mapred-queues.xml. And the system is expected to have atleast one queue - the default queue. This is ensured in a default setup, because there's a default mapred-queues.xml.template that's copied over to become mapred-queues.xml during build.

        However, for backwards compatibility, you could have queues defined in mapred-site.xml. In your case when conf/mapred-queues.xml did not exist, were there queues configured in mapred-site ?

        Show
        Hemanth Yamijala added a comment - Aaron, in Hadoop 0.21, we are shifting the queue configuration to mapred-queues.xml. And the system is expected to have atleast one queue - the default queue. This is ensured in a default setup, because there's a default mapred-queues.xml.template that's copied over to become mapred-queues.xml during build. However, for backwards compatibility, you could have queues defined in mapred-site.xml. In your case when conf/mapred-queues.xml did not exist, were there queues configured in mapred-site ?
        Hide
        Hemanth Yamijala added a comment -

        Also, it is unlikely this is caused by MAPREDUCE-862. If at all, it should be caused by MAPREDUCE-861. This patch did not touch the QueueManager code at all.

        Show
        Hemanth Yamijala added a comment - Also, it is unlikely this is caused by MAPREDUCE-862 . If at all, it should be caused by MAPREDUCE-861 . This patch did not touch the QueueManager code at all.
        Hide
        Aaron Kimball added a comment -

        You're right. I need more caffeine before using git blame in the future. Moving discussion there.

        Show
        Aaron Kimball added a comment - You're right. I need more caffeine before using git blame in the future. Moving discussion there.

          People

          • Assignee:
            V.V.Chaitanya Krishna
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development