Hadoop Common
  1. Hadoop Common
  2. HADOOP-4190

Changes to JobHistory makes it backward incompatible

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Changed job history format to add a dot at end of each line.

      Description

      With HADOOP-3245, the job history format has changed. Every log line now ends with a '.' (dot). JobHistory's parser will consider a history line valid only if it ends in a dot
      Hence the new parser wont be able to parse old history files.

      1. HADOOP-4190-v3.patch
        11 kB
        Amar Kamat
      2. HADOOP-4190-v4.patch
        14 kB
        Amar Kamat
      3. HADOOP-4190-v5.patch
        15 kB
        Amar Kamat
      4. HADOOP-4190-v6.patch
        15 kB
        Amar Kamat
      5. HADOOP-4190-v6.1.patch
        15 kB
        Amar Kamat

        Activity

        Hide
        Amar Kamat added a comment -

        Ideally there should be some kind of a version attached to JobHistory. A parser can parse a history file only if their version matches. Thoughts?

        Show
        Amar Kamat added a comment - Ideally there should be some kind of a version attached to JobHistory. A parser can parse a history file only if their version matches. Thoughts?
        Hide
        Doug Cutting added a comment -

        +1 Adding a format version to the file is definitely a good idea.

        Show
        Doug Cutting added a comment - +1 Adding a format version to the file is definitely a good idea.
        Hide
        Amar Kamat added a comment -

        Attaching a patch that

        • adds version to job history which is logged to the history file as the 1st line
        • switches to old/default regex for line-end matching if the version info is missing else uses the new regex
        • adds a test case that tests the job history parser with old and new file formats

        Manually tested the patch with old and new history files and works fine. More testing in progress. Will try and make it more cleaner and optimized.

        Show
        Amar Kamat added a comment - Attaching a patch that adds version to job history which is logged to the history file as the 1st line switches to old/default regex for line-end matching if the version info is missing else uses the new regex adds a test case that tests the job history parser with old and new file formats Manually tested the patch with old and new history files and works fine. More testing in progress. Will try and make it more cleaner and optimized.
        Hide
        Amar Kamat added a comment -

        Will resubmit the patch that applies to trunk.

        Show
        Amar Kamat added a comment - Will resubmit the patch that applies to trunk.
        Hide
        Arun C Murthy added a comment -

        +1 to the concept.

        As an aside: please ensure this patch works post HADOOP-2165 too. Thanks!

        Show
        Arun C Murthy added a comment - +1 to the concept. As an aside: please ensure this patch works post HADOOP-2165 too. Thanks!
        Hide
        Amar Kamat added a comment -

        Attaching a patch that takes care of the escaping/un-escaping business.

        Show
        Amar Kamat added a comment - Attaching a patch that takes care of the escaping/un-escaping business.
        Hide
        Arun C Murthy added a comment -

        Did this patch pass 'ant test-patch test' ?

        Show
        Arun C Murthy added a comment - Did this patch pass 'ant test-patch test' ?
        Hide
        Arun C Murthy added a comment -

        I ran 'ant test' and TestJobHistoryParsing failed with:

        estsuite: org.apache.hadoop.mapred.TestJobHistoryParsing
        Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.307 sec
        
        Testcase: testHistoryParsing took 0.305 sec
          FAILED
        expected:<Value has equal=to, "quotes" and spaces in it> but was:<>
        junit.framework.ComparisonFailure: expected:<Value has equal=to, "quotes" and spaces in it> but was:<>
          at org.apache.hadoop.mapred.TestJobHistoryParsing.testHistoryParsing(TestJobHistoryParsing.java:93)
        
        Show
        Arun C Murthy added a comment - I ran 'ant test' and TestJobHistoryParsing failed with: estsuite: org.apache.hadoop.mapred.TestJobHistoryParsing Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.307 sec Testcase: testHistoryParsing took 0.305 sec FAILED expected:<Value has equal=to, "quotes" and spaces in it> but was:<> junit.framework.ComparisonFailure: expected:<Value has equal=to, "quotes" and spaces in it> but was:<> at org.apache.hadoop.mapred.TestJobHistoryParsing.testHistoryParsing(TestJobHistoryParsing.java:93)
        Hide
        Arun C Murthy added a comment -

        Forgot to add that this testcase does not fail without this patch...

        Show
        Arun C Murthy added a comment - Forgot to add that this testcase does not fail without this patch...
        Hide
        Amar Kamat added a comment -

        Thanks Arun for the ant-test. The test failed because of missing version line.

        Show
        Amar Kamat added a comment - Thanks Arun for the ant-test. The test failed because of missing version line.
        Hide
        Amar Kamat added a comment -

        Result of test-patch on branch-19

             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
        

        Running ant-test.

        Show
        Amar Kamat added a comment - Result of test-patch on branch-19 [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. Running ant-test.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12390563/HADOOP-4190-v5.patch
        against trunk revision 698187.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3354/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3354/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3354/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3354/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390563/HADOOP-4190-v5.patch against trunk revision 698187. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3354/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3354/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3354/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3354/console This message is automatically generated.
        Hide
        Amar Kamat added a comment -

        Had an off-line discussion with Devaraj. Devaraj suggested that the line delimiter would not change across versions. Hence later versions should also use '.' as the delim. Attaching a patch that incorporates these comments.
        ant test-patch fails on findbugs. Didnt see any thing related to my patch. ant test passed on my box.

        Show
        Amar Kamat added a comment - Had an off-line discussion with Devaraj. Devaraj suggested that the line delimiter would not change across versions. Hence later versions should also use '.' as the delim. Attaching a patch that incorporates these comments. ant test-patch fails on findbugs. Didnt see any thing related to my patch. ant test passed on my box.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12390907/HADOOP-4190-v6.patch
        against trunk revision 698721.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3370/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3370/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3370/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390907/HADOOP-4190-v6.patch against trunk revision 698721. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3370/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3370/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3370/console This message is automatically generated.
        Hide
        Amar Kamat added a comment -

        The findbugs warning is because of the IOException which the parser ignores while it tries to close the stream. Not something that I have introduced. Here is a section from the patch that explains the warning

        -      }
        +      } while ((line = reader.readLine())!= null);
             } finally {
               try { reader.close(); } catch (IOException ex) {}
        
        Show
        Amar Kamat added a comment - The findbugs warning is because of the IOException which the parser ignores while it tries to close the stream. Not something that I have introduced. Here is a section from the patch that explains the warning - } + } while ((line = reader.readLine())!= null ); } finally { try { reader.close(); } catch (IOException ex) {}
        Hide
        Amareshwari Sriramadasu added a comment -

        one minor nit - I think the newly added enums should be added at the end.

        Show
        Amareshwari Sriramadasu added a comment - one minor nit - I think the newly added enums should be added at the end.
        Hide
        Amar Kamat added a comment -

        Attaching an updated patch the incorporates Amareshwari's comment.

        Show
        Amar Kamat added a comment - Attaching an updated patch the incorporates Amareshwari's comment.
        Hide
        Amareshwari Sriramadasu added a comment -

        +1 Latest patch looks good

        Show
        Amareshwari Sriramadasu added a comment - +1 Latest patch looks good
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amar!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amar!
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #618 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/618/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #654 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/654/)
        . Adding testcase that I forgot to add earlier.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #654 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/654/ ) . Adding testcase that I forgot to add earlier.

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Amar Kamat
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development