Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6549

multibyte delimiters with LineRecordReader cause duplicate records

    Details

    • Hadoop Flags:
      Reviewed

      Description

      LineRecorderReader currently produces duplicate records under certain scenarios such as:

      1) input string: "abc++defghi+"
      delimiter string: "+++"
      test passes with all sizes of the split
      2) input string: "abc+def+ghi+"
      delimiter string: "+++"
      test fails with a split size of 4
      2) input string: "abc++defghi+"
      delimiter string: "++"
      test fails with a split size of 5
      3) input string "abc++defghij+"
      delimiter string: "++"
      test fails with a split size of 4
      4) input string "abc+def+ghi+"
      delimiter string: "++"
      test fails with a split size of 9

      1. MAPREDUCE-6549.3.patch
        31 kB
        Wilfred Spiegelenburg
      2. MAPREDUCE-6549-1.patch
        3 kB
        Dustin Cote
      3. MAPREDUCE-6549-2.patch
        30 kB
        Wilfred Spiegelenburg

        Issue Links

          Activity

          Hide
          cotedm Dustin Cote added a comment -

          Attaching a patch to basically remove the attempt to read the last incomplete record of an input and change the tests to test a more generic, imperfect scenario. I'll add some more tests if review deems it necessary. As far as I am aware, we should drop an incomplete record at the end of the input, which now this happens with this patch in addition to the correct number of records coming up in the middle of the input (where previously there were duplicates).

          Show
          cotedm Dustin Cote added a comment - Attaching a patch to basically remove the attempt to read the last incomplete record of an input and change the tests to test a more generic, imperfect scenario. I'll add some more tests if review deems it necessary. As far as I am aware, we should drop an incomplete record at the end of the input, which now this happens with this patch in addition to the correct number of records coming up in the middle of the input (where previously there were duplicates).
          Hide
          cotedm Dustin Cote added a comment -

          zhihai xu, could you review this?

          Show
          cotedm Dustin Cote added a comment - zhihai xu , could you review this?
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          I tried the change that you made in the patch and it fails the current tests.
          The patch changes one test (TestLineRecordReader.java) but we have two versions. The mapred version is unchanged and now fails. The mapreduce version works but as soon as I change the delimiter back it also fails. That means that the change does not fix the issue.

          it also brings the two tests out of sync which is not correct

          Show
          wilfreds Wilfred Spiegelenburg added a comment - I tried the change that you made in the patch and it fails the current tests. The patch changes one test (TestLineRecordReader.java) but we have two versions. The mapred version is unchanged and now fails. The mapreduce version works but as soon as I change the delimiter back it also fails. That means that the change does not fix the issue. it also brings the two tests out of sync which is not correct
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          Dustin Cote I have picked up the jira and have a fully tested and working patch for the issue

          Show
          wilfreds Wilfred Spiegelenburg added a comment - Dustin Cote I have picked up the jira and have a fully tested and working patch for the issue
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          The issue is related to MAPREDUCE-6481. That jira changed the position calculation and made sure that the full records are returned by the reader as expected. It did not anticipate the record duplication. Junit tests also did not cover the use cases correctly to discover the issue.
          The problem is limited to multi byte delimiters only as far as I can trace.

          The junit tests for the multi byte delimiter only take the best case scenario into account. The input data contained the exact delimiter and no ambiguous characters. As soon as the test is changed, either the delimiter or the input data, a failure will be triggered. The issue with the failure is that it does not clearly show when and how it fails. Analysis of the test failures shows that a complex combination of input data, split and buffer size will trigger a failure.

          Based on testing the duplication of the record occurs only if:

          • the first character(s) of the delimiter are part of the record data, example:
            1) the delimiter is += and the data contains a + and is not followed by =
            2) the delimiter is +=+= and the data contains +=+ and is not followed by =
          • the delimiter character is found at the split boundary: the last character before the split ends
          • a fill of the buffer is triggered to finish processing the record

          The underlying problem is that we set a flag called needAdditionalRecord in the UncompressedSplitLineReader when we fill the buffer and have encountered part of a delimiter in combination with a split. We keep track of this in the ambiguous character number. However is it turns out that if the character(s) found after that point do not belong to a delimiter we do not unset the needAdditionalRecord. This causes the next record to be read twice and thus we see a duplication of records.
          The solution would be to unset the flag when we detect that we're not processing a delimiter. We currently only add the ambiguous characters to the record read and set the number back to 0. At the same point we need to unset the flag.

          The patch was developed based on junit tests that exercise the split and buffer settings in combination with multiple delimiter types using different inputs. All cases now provide a consistent count of records and correct position inside the data.

          Show
          wilfreds Wilfred Spiegelenburg added a comment - The issue is related to MAPREDUCE-6481 . That jira changed the position calculation and made sure that the full records are returned by the reader as expected. It did not anticipate the record duplication. Junit tests also did not cover the use cases correctly to discover the issue. The problem is limited to multi byte delimiters only as far as I can trace. The junit tests for the multi byte delimiter only take the best case scenario into account. The input data contained the exact delimiter and no ambiguous characters. As soon as the test is changed, either the delimiter or the input data, a failure will be triggered. The issue with the failure is that it does not clearly show when and how it fails. Analysis of the test failures shows that a complex combination of input data, split and buffer size will trigger a failure. Based on testing the duplication of the record occurs only if: the first character(s) of the delimiter are part of the record data, example: 1) the delimiter is += and the data contains a + and is not followed by = 2) the delimiter is +=+= and the data contains +=+ and is not followed by = the delimiter character is found at the split boundary: the last character before the split ends a fill of the buffer is triggered to finish processing the record The underlying problem is that we set a flag called needAdditionalRecord in the UncompressedSplitLineReader when we fill the buffer and have encountered part of a delimiter in combination with a split. We keep track of this in the ambiguous character number. However is it turns out that if the character(s) found after that point do not belong to a delimiter we do not unset the needAdditionalRecord . This causes the next record to be read twice and thus we see a duplication of records. The solution would be to unset the flag when we detect that we're not processing a delimiter. We currently only add the ambiguous characters to the record read and set the number back to 0. At the same point we need to unset the flag. The patch was developed based on junit tests that exercise the split and buffer settings in combination with multiple delimiter types using different inputs. All cases now provide a consistent count of records and correct position inside the data.
          Hide
          zxu zhihai xu added a comment -

          Nice catch! But I think this issue is not related to MAPREDUCE-6481. Without MAPREDUCE-6481, this issue will still happen. Also I think the same issue may also happen for compressed input. The attached patch only fix the issue for uncompressed input.

          Show
          zxu zhihai xu added a comment - Nice catch! But I think this issue is not related to MAPREDUCE-6481 . Without MAPREDUCE-6481 , this issue will still happen. Also I think the same issue may also happen for compressed input. The attached patch only fix the issue for uncompressed input.
          Hide
          cotedm Dustin Cote added a comment -

          Wilfred Spiegelenburg

          Are you sure your fix is the right one? The reason I changed the tests was because they validate an incomplete record for some reason at the end (which is the part of my fix that is breaking tests in the mapred package since I forgot to change them). The reason I'm saying this is because I would expect the following:
          Input: abcdefghij+kl+mno
          Output records: 1) abcdefghij 2) kl

          It looks like your tests do the same thing. mno doesn't have a delimiter at the end, so isn't that garbage data as an incomplete record? I would expect that to be the behavior if I were using the API, but I don't see any real documentation for this on multibyte delimiters. If we're going to commit to the last part of the data without a delimiter at the end as being a record, then that should be documented as well. Otherwise, I'd rather merge our patches together and verify the functionality so that the above scenario is what happens instead of pulling in undelimited data at the end of the file.

          Show
          cotedm Dustin Cote added a comment - Wilfred Spiegelenburg Are you sure your fix is the right one? The reason I changed the tests was because they validate an incomplete record for some reason at the end (which is the part of my fix that is breaking tests in the mapred package since I forgot to change them). The reason I'm saying this is because I would expect the following: Input: abcdefghij+ kl +mno Output records: 1) abcdefghij 2) kl It looks like your tests do the same thing. mno doesn't have a delimiter at the end, so isn't that garbage data as an incomplete record? I would expect that to be the behavior if I were using the API, but I don't see any real documentation for this on multibyte delimiters. If we're going to commit to the last part of the data without a delimiter at the end as being a record, then that should be documented as well. Otherwise, I'd rather merge our patches together and verify the functionality so that the above scenario is what happens instead of pulling in undelimited data at the end of the file.
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          zhihai xu The change in MAPREDUCE-6481 is not to blame for the duplicate records as far as I can tell. It fixed things and now we get to see what is there and thus we noticed the duplicates. I did not look at the compressed input, and I do think you are correct. Compressed input uses the same steps and we should clear the setting in the same way as we did the uncompressed stream. I will try to generate a compressed stream that is splittable to get a test case. I will upload a new patch but I will first try to generate the test case before I do that.

          Dustin Cote An EOF will automatically terminate the record there is no need for a record delimiter at the end of the file. All the test, and comments in the code show it. The assumption is that the last record before EOF does not need a record terminator. It is not a new assumption, assuming that an EOF would not delimit a record would be counter intuitive. Most text files for instance do not have a newline at the last line.

          Show
          wilfreds Wilfred Spiegelenburg added a comment - zhihai xu The change in MAPREDUCE-6481 is not to blame for the duplicate records as far as I can tell. It fixed things and now we get to see what is there and thus we noticed the duplicates. I did not look at the compressed input, and I do think you are correct. Compressed input uses the same steps and we should clear the setting in the same way as we did the uncompressed stream. I will try to generate a compressed stream that is splittable to get a test case. I will upload a new patch but I will first try to generate the test case before I do that. Dustin Cote An EOF will automatically terminate the record there is no need for a record delimiter at the end of the file. All the test, and comments in the code show it. The assumption is that the last record before EOF does not need a record terminator. It is not a new assumption, assuming that an EOF would not delimit a record would be counter intuitive. Most text files for instance do not have a newline at the last line.
          Hide
          cotedm Dustin Cote added a comment -

          Wilfred Spiegelenburg ok I agree that it works this way, but I disagree on the assumption that text files don't usually end with a new line on the last line. At least POSIX standards (which I consider intuitive) define lines to require a newline at the end --> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206

          That said, it looks like LineRecordReader is indeed always using EOF as something equivalent to a delimiter, so it probably makes more sense to stick with the convention defined with the LineRecordReader's defaults already.

          Show
          cotedm Dustin Cote added a comment - Wilfred Spiegelenburg ok I agree that it works this way, but I disagree on the assumption that text files don't usually end with a new line on the last line. At least POSIX standards (which I consider intuitive) define lines to require a newline at the end --> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206 That said, it looks like LineRecordReader is indeed always using EOF as something equivalent to a delimiter, so it probably makes more sense to stick with the convention defined with the LineRecordReader's defaults already.
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          I have been able to generate a compressed file which shows a same record duplication as was shown in the uncompressed processing. The code however behaves completely different in the two cases since we do not have the same kind of buffer filling process. I am still trying to fix the compressed code without breaking the uncompressed code.

          I should have a fix for both cases in a day or two.

          Show
          wilfreds Wilfred Spiegelenburg added a comment - I have been able to generate a compressed file which shows a same record duplication as was shown in the uncompressed processing. The code however behaves completely different in the two cases since we do not have the same kind of buffer filling process. I am still trying to fix the compressed code without breaking the uncompressed code. I should have a fix for both cases in a day or two.
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          The compressed version is not as easily fixable, and I am opening up a new jira for that one.

          The compressed version does not use the split size as the uncompressed version does. The split size as far as I can tell depends on the compression codec and the file encoding/compression blocks. The split size is not taken into account as it is in the uncompressed version.

          I ran a set of similar junit tests over the compressed data and the changed code is not even triggered.

          Show
          wilfreds Wilfred Spiegelenburg added a comment - The compressed version is not as easily fixable, and I am opening up a new jira for that one. The compressed version does not use the split size as the uncompressed version does. The split size as far as I can tell depends on the compression codec and the file encoding/compression blocks. The split size is not taken into account as it is in the uncompressed version. I ran a set of similar junit tests over the compressed data and the changed code is not even triggered.
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          zhihai xu & Jason Lowe can you also please have a look at the patch for the uncompressed version?
          I have not seen a build being triggered from the patch that was added. That might need to be triggered somehow for this patch (naming convention wrong for the patch?)

          Show
          wilfreds Wilfred Spiegelenburg added a comment - zhihai xu & Jason Lowe can you also please have a look at the patch for the uncompressed version? I have not seen a build being triggered from the patch that was added. That might need to be triggered somehow for this patch (naming convention wrong for the patch?)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 43s trunk passed
          +1 compile 10m 12s trunk passed with JDK v1.8.0_66
          +1 compile 10m 9s trunk passed with JDK v1.7.0_85
          +1 checkstyle 1m 8s trunk passed
          +1 mvnsite 1m 46s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          -1 findbugs 2m 11s hadoop-common-project/hadoop-common in trunk has 1 extant Findbugs warnings.
          -1 findbugs 1m 20s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 42s trunk passed with JDK v1.7.0_85
          +1 mvninstall 2m 14s the patch passed
          +1 compile 10m 19s the patch passed with JDK v1.8.0_66
          +1 javac 10m 19s the patch passed
          +1 compile 10m 5s the patch passed with JDK v1.7.0_85
          +1 javac 10m 5s the patch passed
          +1 checkstyle 1m 9s the patch passed
          +1 mvnsite 1m 42s the patch passed
          +1 mvneclipse 0m 30s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 46s the patch passed
          +1 javadoc 1m 30s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 41s the patch passed with JDK v1.7.0_85
          -1 unit 7m 38s hadoop-common in the patch failed with JDK v1.8.0_66.
          -1 unit 2m 19s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_66.
          -1 unit 7m 26s hadoop-common in the patch failed with JDK v1.7.0_85.
          -1 unit 2m 31s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 23s Patch generated 1 ASF License warnings.
          93m 59s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.fs.permission.TestFsPermission
            hadoop.mapreduce.lib.input.TestLineRecordReader
          JDK v1.7.0_85 Failed junit tests hadoop.ipc.TestDecayRpcScheduler
            hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.fs.permission.TestFsPermission
            hadoop.mapreduce.lib.input.TestLineRecordReader



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772422/MAPREDUCE-6549-2.patch
          JIRA Issue MAPREDUCE-6549
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7b2604a891e5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 28dfe72
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 43s trunk passed +1 compile 10m 12s trunk passed with JDK v1.8.0_66 +1 compile 10m 9s trunk passed with JDK v1.7.0_85 +1 checkstyle 1m 8s trunk passed +1 mvnsite 1m 46s trunk passed +1 mvneclipse 0m 31s trunk passed -1 findbugs 2m 11s hadoop-common-project/hadoop-common in trunk has 1 extant Findbugs warnings. -1 findbugs 1m 20s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 42s trunk passed with JDK v1.7.0_85 +1 mvninstall 2m 14s the patch passed +1 compile 10m 19s the patch passed with JDK v1.8.0_66 +1 javac 10m 19s the patch passed +1 compile 10m 5s the patch passed with JDK v1.7.0_85 +1 javac 10m 5s the patch passed +1 checkstyle 1m 9s the patch passed +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 46s the patch passed +1 javadoc 1m 30s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 41s the patch passed with JDK v1.7.0_85 -1 unit 7m 38s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 2m 19s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_66. -1 unit 7m 26s hadoop-common in the patch failed with JDK v1.7.0_85. -1 unit 2m 31s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 23s Patch generated 1 ASF License warnings. 93m 59s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.permission.TestFsPermission   hadoop.mapreduce.lib.input.TestLineRecordReader JDK v1.7.0_85 Failed junit tests hadoop.ipc.TestDecayRpcScheduler   hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.fs.permission.TestFsPermission   hadoop.mapreduce.lib.input.TestLineRecordReader Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772422/MAPREDUCE-6549-2.patch JIRA Issue MAPREDUCE-6549 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7b2604a891e5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 28dfe72 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6186/console This message was automatically generated.
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          Updated the patch to fix the NPE in the testUncompressedInputCustomDelimiterPosValue

          Checked the license, findbugs and other junit test failures and they are not related to the changes from this patch

          Show
          wilfreds Wilfred Spiegelenburg added a comment - Updated the patch to fix the NPE in the testUncompressedInputCustomDelimiterPosValue Checked the license, findbugs and other junit test failures and they are not related to the changes from this patch
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 46s trunk passed
          +1 compile 9m 40s trunk passed with JDK v1.8.0_66
          +1 compile 9m 51s trunk passed with JDK v1.7.0_85
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 1m 50s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          -1 findbugs 1m 57s hadoop-common-project/hadoop-common in trunk has 1 extant Findbugs warnings.
          -1 findbugs 1m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 1m 23s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 37s trunk passed with JDK v1.7.0_85
          +1 mvninstall 2m 16s the patch passed
          +1 compile 8m 40s the patch passed with JDK v1.8.0_66
          +1 javac 8m 40s the patch passed
          +1 compile 8m 38s the patch passed with JDK v1.7.0_85
          +1 javac 8m 38s the patch passed
          +1 checkstyle 1m 0s the patch passed
          +1 mvnsite 1m 37s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 12s the patch passed
          +1 javadoc 1m 17s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 31s the patch passed with JDK v1.7.0_85
          -1 unit 7m 7s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 1m 51s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          -1 unit 7m 15s hadoop-common in the patch failed with JDK v1.7.0_85.
          +1 unit 2m 13s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_85.
          -1 asflicense 0m 22s Patch generated 1 ASF License warnings.
          86m 42s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.fs.permission.TestFsPermission
          JDK v1.7.0_85 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.ipc.TestIPC
            hadoop.fs.permission.TestFsPermission



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774214/MAPREDUCE-6549.3.patch
          JIRA Issue MAPREDUCE-6549
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dac75dfa74ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 23c625e
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 46s trunk passed +1 compile 9m 40s trunk passed with JDK v1.8.0_66 +1 compile 9m 51s trunk passed with JDK v1.7.0_85 +1 checkstyle 1m 6s trunk passed +1 mvnsite 1m 50s trunk passed +1 mvneclipse 0m 29s trunk passed -1 findbugs 1m 57s hadoop-common-project/hadoop-common in trunk has 1 extant Findbugs warnings. -1 findbugs 1m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 1m 23s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 37s trunk passed with JDK v1.7.0_85 +1 mvninstall 2m 16s the patch passed +1 compile 8m 40s the patch passed with JDK v1.8.0_66 +1 javac 8m 40s the patch passed +1 compile 8m 38s the patch passed with JDK v1.7.0_85 +1 javac 8m 38s the patch passed +1 checkstyle 1m 0s the patch passed +1 mvnsite 1m 37s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 12s the patch passed +1 javadoc 1m 17s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 31s the patch passed with JDK v1.7.0_85 -1 unit 7m 7s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 1m 51s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. -1 unit 7m 15s hadoop-common in the patch failed with JDK v1.7.0_85. +1 unit 2m 13s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_85. -1 asflicense 0m 22s Patch generated 1 ASF License warnings. 86m 42s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.permission.TestFsPermission JDK v1.7.0_85 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.ipc.TestIPC   hadoop.fs.permission.TestFsPermission Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774214/MAPREDUCE-6549.3.patch JIRA Issue MAPREDUCE-6549 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dac75dfa74ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23c625e findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6190/console This message was automatically generated.
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          Test failures are not related and tracked in different jiras:

          testIpcWithReaderQueuing is tracked by HADOOP-10406
          testGangliaMetrics2 is tracked in HADOOP-12588
          testDeprecatedUmask is tracked in HDFS-9451

          Show
          wilfreds Wilfred Spiegelenburg added a comment - Test failures are not related and tracked in different jiras: testIpcWithReaderQueuing is tracked by HADOOP-10406 testGangliaMetrics2 is tracked in HADOOP-12588 testDeprecatedUmask is tracked in HDFS-9451
          Hide
          rkanter Robert Kanter added a comment -

          +1 LGTM

          Will commit later today if nobody has any other comments.

          Show
          rkanter Robert Kanter added a comment - +1 LGTM Will commit later today if nobody has any other comments.
          Hide
          rchiang Ray Chiang added a comment -

          +1 (nonbinding)

          Much easier to understand with all the unit tests.

          Show
          rchiang Ray Chiang added a comment - +1 (nonbinding) Much easier to understand with all the unit tests.
          Hide
          rkanter Robert Kanter added a comment -

          Thanks Wilfred and everyone who helped out on this.

          Committed to trunk and branch-2!

          Show
          rkanter Robert Kanter added a comment - Thanks Wilfred and everyone who helped out on this. Committed to trunk and branch-2!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8894/)
          MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8894/ ) MAPREDUCE-6549 . multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1457 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1457/)
          MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1457 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1457/ ) MAPREDUCE-6549 . multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #645 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/645/)
          MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #645 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/645/ ) MAPREDUCE-6549 . multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #726 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/726/)
          MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #726 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/726/ ) MAPREDUCE-6549 . multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2668 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2668/)
          MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2668 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2668/ ) MAPREDUCE-6549 . multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #737 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/737/)
          MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #737 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/737/ ) MAPREDUCE-6549 . multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2582 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2582/)
          MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2582 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2582/ ) MAPREDUCE-6549 . multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          Should this be pulled back into 2.7.3 and 2.6.3 based on the fact that Jason Lowe pulled MAPREDUCE-6481 into those releases?

          Show
          wilfreds Wilfred Spiegelenburg added a comment - Should this be pulled back into 2.7.3 and 2.6.3 based on the fact that Jason Lowe pulled MAPREDUCE-6481 into those releases?
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Wilfred Spiegelenburg! Agree this should be in 2.7.3 and 2.6.3, so I committed this to branch-2.7 and branch-2.6 as well.

          Show
          jlowe Jason Lowe added a comment - Thanks, Wilfred Spiegelenburg ! Agree this should be in 2.7.3 and 2.6.3, so I committed this to branch-2.7 and branch-2.6 as well.
          Hide
          djp Junping Du added a comment -

          Hi Wilfred Spiegelenburg, Robert Kanter and Jason Lowe, I noticed that MAPREDUCE-6558 (compressed version) is still open. Do we think this patch could go independently to 2.6.3 or better to be combined with MAPREDUCE-6558 to 2.6.4?

          Show
          djp Junping Du added a comment - Hi Wilfred Spiegelenburg , Robert Kanter and Jason Lowe , I noticed that MAPREDUCE-6558 (compressed version) is still open. Do we think this patch could go independently to 2.6.3 or better to be combined with MAPREDUCE-6558 to 2.6.4?
          Hide
          jlowe Jason Lowe added a comment -

          It can go independently, which is why I already put it into 2.6.3 per the above comment. It fixes the uncompressed input data case, whereas MAPREDUCE-6558 is tracking the issue with compressed input.

          Show
          jlowe Jason Lowe added a comment - It can go independently, which is why I already put it into 2.6.3 per the above comment. It fixes the uncompressed input data case, whereas MAPREDUCE-6558 is tracking the issue with compressed input.
          Hide
          djp Junping Du added a comment -

          Sounds good to have it in 2.6.3. Thanks Jason!

          Show
          djp Junping Du added a comment - Sounds good to have it in 2.6.3. Thanks Jason!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9102 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9102/)
          MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (vinodkv: rev 9b33a03900c9ffdb9f4b4c59161fdce39ca8c73d)

          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9102 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9102/ ) MAPREDUCE-6549 . multibyte delimiters with LineRecordReader cause (vinodkv: rev 9b33a03900c9ffdb9f4b4c59161fdce39ca8c73d) hadoop-mapreduce-project/CHANGES.txt

            People

            • Assignee:
              wilfreds Wilfred Spiegelenburg
              Reporter:
              cotedm Dustin Cote
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development