Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3166

Make Rumen use job history api instead of relying on current history file name format

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: tools/rumen
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Makes Rumen use job history api instead of relying on current history file name format.

      Description

      Rumen should not depend on the regular expression of job history file name format and should use the newly added api like isValidJobHistoryFileName(), getJobIDFromHistoryFilePath().

      1. MR3166.patch
        5 kB
        Ravi Gummadi

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #58 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/58/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk)

        amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #58 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/58/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #57 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/57/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk)

        amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #57 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/57/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #122 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/122/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk)

        amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #122 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/122/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #113 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/113/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk)

        amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #113 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/113/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #112 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/112/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk)

        amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #112 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/112/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. (Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195540 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Hide
        Amar Kamat added a comment -

        Committed this to branch-0.23.

        Show
        Amar Kamat added a comment - Committed this to branch-0.23.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #829 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/829/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format.

        ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #829 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/829/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #858 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/858/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format.

        ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #858 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/858/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1082 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1082/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format.

        ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1082 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1082/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1140 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1140/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format.

        ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1140 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1140/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1062 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1062/)
        MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying on current history file name format.

        ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1062 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1062/ ) MAPREDUCE-3166 . [Rumen] Make Rumen use job history api instead of relying on current history file name format. ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
        Ravi Gummadi made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Ravi Gummadi added a comment -

        I just committed this to trunk.

        Show
        Ravi Gummadi added a comment - I just committed this to trunk.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12498577/MR3166.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/994//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/994//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498577/MR3166.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/994//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/994//console This message is automatically generated.
        Ravi Gummadi made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags Reviewed [ 10343 ]
        Release Note Makes Rumen use job history api instead of relying on current history file name format.
        Hide
        Ravi Gummadi added a comment -

        Unit tests of Rumen and Gridmix passed on my local machine except TestCompressionEmulationUtils which is a known issue without this patch also.

        test-patch along with MR-3157 gave the following:

        -1 overall.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        I don't see any issues in patchReleaseAuditProblems.txt related to these patches.

        Show
        Ravi Gummadi added a comment - Unit tests of Rumen and Gridmix passed on my local machine except TestCompressionEmulationUtils which is a known issue without this patch also. test-patch along with MR-3157 gave the following: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). I don't see any issues in patchReleaseAuditProblems.txt related to these patches.
        Hide
        Amar Kamat added a comment -

        +1. Looks good to me.

        Show
        Amar Kamat added a comment - +1. Looks good to me.
        Ravi Gummadi made changes -
        Field Original Value New Value
        Attachment MR3166.patch [ 12498577 ]
        Hide
        Ravi Gummadi added a comment -

        Attaching patch on top of patch of MR3157.

        Show
        Ravi Gummadi added a comment - Attaching patch on top of patch of MR3157.
        Ravi Gummadi created issue -

          People

          • Assignee:
            Ravi Gummadi
            Reporter:
            Ravi Gummadi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development