Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3202

Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0

    Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: trunk
    • Component/s: jobhistoryserver
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hadoop Vaidya analysis available using Job History Server UI in 2.0 branch. Also ability for users to add new rules is provided.

      Description

      Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch.

      1. MAPREDUCE-3202.patch
        190 kB
        vitthal (Suhas) Gogate
      2. MAPREDUCE-3202.patch
        188 kB
        vitthal (Suhas) Gogate

        Activity

        Hide
        vitthal (Suhas) Gogate added a comment -

        Submitting the patch to make Hadoop Vaidya work with 2.0 branch and also integrate it with Job History server providing UI interface analyze M/R jobs using Vaidya tool.

        Thanks to Evan Meng @ Pivotal Inc Evan (Yiming) Meng make this happen. Submitting the patch on his behalf.

        Show
        vitthal (Suhas) Gogate added a comment - Submitting the patch to make Hadoop Vaidya work with 2.0 branch and also integrate it with Job History server providing UI interface analyze M/R jobs using Vaidya tool. Thanks to Evan Meng @ Pivotal Inc Evan (Yiming) Meng make this happen. Submitting the patch on his behalf.
        Hide
        vitthal (Suhas) Gogate added a comment -

        Will submit the patch for trunk soon.

        Show
        vitthal (Suhas) Gogate added a comment - Will submit the patch for trunk soon.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12605914/vaidya-2.0.6-alpha.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4062//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605914/vaidya-2.0.6-alpha.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4062//console This message is automatically generated.
        Hide
        vitthal (Suhas) Gogate added a comment -

        Submitting patch for trunk.

        Show
        vitthal (Suhas) Gogate added a comment - Submitting patch for trunk.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12605989/MAPREDUCE-3202.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified test files.

        -1 javac. The applied patch generated 1536 javac compiler warnings (more than the trunk's current 1534 warnings).

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya:

        org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html
        Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/diffJavacWarnings.txt
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605989/MAPREDUCE-3202.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified test files. -1 javac . The applied patch generated 1536 javac compiler warnings (more than the trunk's current 1534 warnings). +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya: org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//console This message is automatically generated.
        Hide
        vitthal (Suhas) Gogate added a comment -

        attaching new patch fixing Hadoop QA warnings

        Show
        vitthal (Suhas) Gogate added a comment - attaching new patch fixing Hadoop QA warnings
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12606157/MAPREDUCE-3202.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya:

        org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606157/MAPREDUCE-3202.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya: org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4074//console This message is automatically generated.
        Hide
        vitthal (Suhas) Gogate added a comment -

        Fixing unit-test failures.

        Show
        vitthal (Suhas) Gogate added a comment - Fixing unit-test failures.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12607659/MAPREDUCE-3202.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607659/MAPREDUCE-3202.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4109//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12607698/MAPREDUCE-3202.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4111//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4111//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607698/MAPREDUCE-3202.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4111//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4111//console This message is automatically generated.
        Hide
        vitthal (Suhas) Gogate added a comment -

        Can I request one of the Yarn/MapReduce committers to review and commit the vaidya patch? It is a contrib project. Thanks in advance! --Suhas

        Show
        vitthal (Suhas) Gogate added a comment - Can I request one of the Yarn/MapReduce committers to review and commit the vaidya patch? It is a contrib project. Thanks in advance! --Suhas
        Hide
        Roman Shaposhnik added a comment -

        vitthal (Suhas) Gogate I did the first pass over the patch and it looks reasonable. Here's a few bits of feedback I've got so far:

        • I would like to suggest that an extra JIRA (perhaps a subjira of this one) gets created to track just the changes required in existing HS webUI to support Vaidya. One half of the patch will go there.
        • please make sure to add unit tests for that first half of the patch
        • on this JIRA we shall keep track of things under hadoop-tools/hadoop-vaidya
        • we would also need to make sure that hadoop-tools/hadoop-vaidya bits end up in maven assembly
        Show
        Roman Shaposhnik added a comment - vitthal (Suhas) Gogate I did the first pass over the patch and it looks reasonable. Here's a few bits of feedback I've got so far: I would like to suggest that an extra JIRA (perhaps a subjira of this one) gets created to track just the changes required in existing HS webUI to support Vaidya. One half of the patch will go there. please make sure to add unit tests for that first half of the patch on this JIRA we shall keep track of things under hadoop-tools/hadoop-vaidya we would also need to make sure that hadoop-tools/hadoop-vaidya bits end up in maven assembly

          People

          • Assignee:
            Evan (Yiming) Meng
            Reporter:
            vitthal (Suhas) Gogate
          • Votes:
            1 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Created:
              Updated:

              Development