Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2802

[MR-279] Jobhistory filenames should have jobID to help in better parsing

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      For jobID such as job_1312933838300_0007, jobhistory file names are named as job%5F1312933838300%5F0007_<submit_time>ramya<jobname>_<finish_time>_1_1_SUCCEEDED.jhist It would be easier for parsing if the jobIDs were a part of the filenames.

      1. MAPREDUCE-2802.patch
        10 kB
        Jonathan Eagles

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #854 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/854/)
        MAPREDUCE-2802. Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #854 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/854/ ) MAPREDUCE-2802 . Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #40 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/40/)
        Merge -c 1180220 from trunk to branch-0.23 to fix MAPREDUCE-2802.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180222
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #40 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/40/ ) Merge -c 1180220 from trunk to branch-0.23 to fix MAPREDUCE-2802 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180222 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #33 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/33/)
        Merge -c 1180220 from trunk to branch-0.23 to fix MAPREDUCE-2802.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180222
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #33 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/33/ ) Merge -c 1180220 from trunk to branch-0.23 to fix MAPREDUCE-2802 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180222 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #824 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/824/)
        MAPREDUCE-2802. Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #824 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/824/ ) MAPREDUCE-2802 . Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1060 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1060/)
        MAPREDUCE-2802. Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1060 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1060/ ) MAPREDUCE-2802 . Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Hide
        Jonathan Eagles added a comment -

        Thanks so much, Arun!

        Show
        Jonathan Eagles added a comment - Thanks so much, Arun!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1041 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1041/)
        MAPREDUCE-2802. Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1041 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1041/ ) MAPREDUCE-2802 . Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1119 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1119/)
        MAPREDUCE-2802. Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1119 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1119/ ) MAPREDUCE-2802 . Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks Jonathan!

        (Sid - thanks for the review too!)

        Show
        Arun C Murthy added a comment - I just committed this. Thanks Jonathan! (Sid - thanks for the review too!)
        Hide
        Jonathan Eagles added a comment -

        Thanks, Sid!

        Show
        Jonathan Eagles added a comment - Thanks, Sid!
        Hide
        Siddharth Seth added a comment -

        lgtm. NB +1.
        Changing the delimiter may be an issue for some apps - but the .23 jobhistory filename has anyway changed, and there are APIs available to pull information from the filename.

        Show
        Siddharth Seth added a comment - lgtm. NB +1. Changing the delimiter may be an issue for some apps - but the .23 jobhistory filename has anyway changed, and there are APIs available to pull information from the filename.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12498078/MAPREDUCE-2802.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/961//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/961//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498078/MAPREDUCE-2802.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/961//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/961//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        +1 overall.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        Jonathan Eagles added a comment - +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Jonathan Eagles added a comment -

        The patch adds a new test file and directory.

        Show
        Jonathan Eagles added a comment - The patch adds a new test file and directory.
        Hide
        Jonathan Eagles added a comment -

        For explanations on why job name "word count" is encoded as "word+count" and safe characters please see the following link.

        http://download.oracle.com/javase/6/docs/api/java/net/URLEncoder.html

        Show
        Jonathan Eagles added a comment - For explanations on why job name "word count" is encoded as "word+count" and safe characters please see the following link. http://download.oracle.com/javase/6/docs/api/java/net/URLEncoder.html
        Hide
        Jonathan Eagles added a comment -

        jobids, user names, and job names are sanitized when generating the job history file name. Since the name of the file needs to be parsed in the current design, a delimiter, underscore '_', was chosen. When an underscore occurs in the job id, user name, or job name it is changed to %5F like you are seeing above.

        Very simply I can change the delimiter used to lessen the likelihood of percent encoding to happen. For example, dash '-' would look like this.

        job_1317928501754_0001-1317928742025-jeagles-word+count-1317928754958-1-1-SUCCEEDED.jhist

        Show
        Jonathan Eagles added a comment - jobids, user names, and job names are sanitized when generating the job history file name. Since the name of the file needs to be parsed in the current design, a delimiter, underscore '_', was chosen. When an underscore occurs in the job id, user name, or job name it is changed to %5F like you are seeing above. Very simply I can change the delimiter used to lessen the likelihood of percent encoding to happen. For example, dash '-' would look like this. job_1317928501754_0001-1317928742025-jeagles-word+count-1317928754958-1-1-SUCCEEDED.jhist

          People

          • Assignee:
            Jonathan Eagles
            Reporter:
            Ramya Sunil
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development