Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3202

Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0

    Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: jobhistoryserver
    • Labels:
    • Target Version/s:
    • Release Note:
      Hadoop Vaidya analysis available using Job History Server UI in 2.0 branch. Also ability for users to add new rules is provided.

      Description

      Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch.

      1. MAPREDUCE-3202.patch
        188 kB
        vitthal (Suhas) Gogate
      2. MAPREDUCE-3202.patch
        190 kB
        vitthal (Suhas) Gogate

        Activity

        vitthal (Suhas) Gogate created issue -
        vitthal (Suhas) Gogate made changes -
        Field Original Value New Value
        Affects Version/s 0.20.205.0 [ 12316391 ]
        vitthal (Suhas) Gogate made changes -
        Summary Integrating Hadoop Vaidya with JobHistory Server Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
        vitthal (Suhas) Gogate made changes -
        Description At present jobdetailshistory page served by JobHistory Server provides elementary job analysis through link "Analyze This job". Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. Integrating Hadoop Vaidya with JobHistory server would really improve the usability of this tool and also benefit many naive users understanding various performance problems and/or best practices violations associated with their job.

        Integration would also aim at providing users a convenient interface where they can manage the existing rules as well as write their own new rules.

        During my tenure at Yahoo, Vaidya tool was successfully deployed in production analyzing tens of thousands of jobs every day with lot more useful rules than the sample ones present in the contrib project. Many of these rules are open sourced already (big thanks to Yahoo! MAPREDUCE-1530) but yet to integrate with the tool.

        I will add more design details for this feature in near future as work towards getting prototype running.. Any thoughts/comments are welcome.
        Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch.
        vitthal (Suhas) Gogate made changes -
        Affects Version/s 2.0.0-alpha [ 12320354 ]
        Affects Version/s 0.20.205.0 [ 12316391 ]
        Affects Version/s 1.0.0 [ 12318240 ]
        vitthal (Suhas) Gogate made changes -
        Attachment vaidya-2.0.6-alpha.patch [ 12605914 ]
        Hide
        vitthal (Suhas) Gogate added a comment -

        Submitting the patch to make Hadoop Vaidya work with 2.0 branch and also integrate it with Job History server providing UI interface analyze M/R jobs using Vaidya tool.

        Thanks to Evan Meng @ Pivotal Inc Evan (Yiming) Meng make this happen. Submitting the patch on his behalf.

        Show
        vitthal (Suhas) Gogate added a comment - Submitting the patch to make Hadoop Vaidya work with 2.0 branch and also integrate it with Job History server providing UI interface analyze M/R jobs using Vaidya tool. Thanks to Evan Meng @ Pivotal Inc Evan (Yiming) Meng make this happen. Submitting the patch on his behalf.
        vitthal (Suhas) Gogate made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note Hadoop Vaidya analysis available using Job History Server UI in 2.0 branch. Also ability for users to add new rules is provided.
        Fix Version/s 2.0.6-alpha [ 12324812 ]
        Hide
        vitthal (Suhas) Gogate added a comment -

        Will submit the patch for trunk soon.

        Show
        vitthal (Suhas) Gogate added a comment - Will submit the patch for trunk soon.
        vitthal (Suhas) Gogate made changes -
        Assignee vitthal (Suhas) Gogate [ vitthal_gogate ]
        vitthal (Suhas) Gogate made changes -
        Assignee vitthal (Suhas) Gogate [ vitthal_gogate ] Evan (Yiming) Meng [ evanmeng ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12605914/vaidya-2.0.6-alpha.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4062//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605914/vaidya-2.0.6-alpha.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4062//console This message is automatically generated.
        vitthal (Suhas) Gogate made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Target Version/s trunk [ 12320360 ]
        Hide
        vitthal (Suhas) Gogate added a comment -

        Submitting patch for trunk.

        Show
        vitthal (Suhas) Gogate added a comment - Submitting patch for trunk.
        vitthal (Suhas) Gogate made changes -
        Attachment MAPREDUCE-3202.patch [ 12605989 ]
        vitthal (Suhas) Gogate made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s trunk [ 12320360 ]
        Fix Version/s 2.0.6-alpha [ 12324812 ]
        vitthal (Suhas) Gogate made changes -
        Attachment vaidya-2.0.6-alpha.patch [ 12605914 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12605989/MAPREDUCE-3202.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified test files.

        -1 javac. The applied patch generated 1536 javac compiler warnings (more than the trunk's current 1534 warnings).

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya:

        org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html
        Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/diffJavacWarnings.txt
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605989/MAPREDUCE-3202.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified test files. -1 javac . The applied patch generated 1536 javac compiler warnings (more than the trunk's current 1534 warnings). +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya: org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//console This message is automatically generated.
        Hide
        vitthal (Suhas) Gogate added a comment -

        attaching new patch fixing Hadoop QA warnings

        Show
        vitthal (Suhas) Gogate added a comment - attaching new patch fixing Hadoop QA warnings
        vitthal (Suhas) Gogate made changes -
        Attachment MAPREDUCE-3202.patch [ 12606157 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12606157/MAPREDUCE-3202.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya:

        org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606157/MAPREDUCE-3202.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya: org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//console This message is automatically generated.
        Hide
        vitthal (Suhas) Gogate added a comment -

        Fixing unit-test failures.

        Show
        vitthal (Suhas) Gogate added a comment - Fixing unit-test failures.
        vitthal (Suhas) Gogate made changes -
        Attachment MAPREDUCE-3202.patch [ 12607659 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12607659/MAPREDUCE-3202.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607659/MAPREDUCE-3202.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//console This message is automatically generated.
        vitthal (Suhas) Gogate made changes -
        Attachment MAPREDUCE-3202.patch [ 12607659 ]
        vitthal (Suhas) Gogate made changes -
        Attachment MAPREDUCE-3202.patch [ 12606157 ]
        vitthal (Suhas) Gogate made changes -
        Attachment MAPREDUCE-3202.patch [ 12607698 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12607698/MAPREDUCE-3202.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4111//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4111//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607698/MAPREDUCE-3202.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4111//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4111//console This message is automatically generated.
        Hide
        vitthal (Suhas) Gogate added a comment -

        Can I request one of the Yarn/MapReduce committers to review and commit the vaidya patch? It is a contrib project. Thanks in advance! --Suhas

        Show
        vitthal (Suhas) Gogate added a comment - Can I request one of the Yarn/MapReduce committers to review and commit the vaidya patch? It is a contrib project. Thanks in advance! --Suhas
        Hide
        Roman Shaposhnik added a comment -

        vitthal (Suhas) Gogate I did the first pass over the patch and it looks reasonable. Here's a few bits of feedback I've got so far:

        • I would like to suggest that an extra JIRA (perhaps a subjira of this one) gets created to track just the changes required in existing HS webUI to support Vaidya. One half of the patch will go there.
        • please make sure to add unit tests for that first half of the patch
        • on this JIRA we shall keep track of things under hadoop-tools/hadoop-vaidya
        • we would also need to make sure that hadoop-tools/hadoop-vaidya bits end up in maven assembly
        Show
        Roman Shaposhnik added a comment - vitthal (Suhas) Gogate I did the first pass over the patch and it looks reasonable. Here's a few bits of feedback I've got so far: I would like to suggest that an extra JIRA (perhaps a subjira of this one) gets created to track just the changes required in existing HS webUI to support Vaidya. One half of the patch will go there. please make sure to add unit tests for that first half of the patch on this JIRA we shall keep track of things under hadoop-tools/hadoop-vaidya we would also need to make sure that hadoop-tools/hadoop-vaidya bits end up in maven assembly
        vitthal (Suhas) Gogate made changes -
        Assignee Evan (Yiming) Meng [ evanmeng ] vitthal (Suhas) Gogate [ vitthal_gogate ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12607698/MAPREDUCE-3202.patch
        against trunk revision d71d40a.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4975//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607698/MAPREDUCE-3202.patch against trunk revision d71d40a. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4975//console This message is automatically generated.
        Allen Wittenauer made changes -
        Fix Version/s trunk [ 12320360 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12607698/MAPREDUCE-3202.patch
        against trunk revision 7711049.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5271//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607698/MAPREDUCE-3202.patch against trunk revision 7711049. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5271//console This message is automatically generated.
        Allen Wittenauer made changes -
        Labels BB2015-05-TBR
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        6h 16m 1 vitthal (Suhas) Gogate 30/Sep/13 23:19
        Open Open Patch Available Patch Available
        712d 21h 46m 2 vitthal (Suhas) Gogate 30/Sep/13 23:23

          People

          • Assignee:
            vitthal (Suhas) Gogate
            Reporter:
            vitthal (Suhas) Gogate
          • Votes:
            2 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Created:
              Updated:

              Development