Hadoop Common
  1. Hadoop Common
  2. HADOOP-8655

In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character sequence in delimiter were found missing in certain cases in the Map Output

    Details

    • Tags:
      hadoop, mapreduce
    • Target Version/s:

      Description

      Set textinputformat.record.delimiter as "</entity>"

      Suppose the input is a text file with the following content
      <entity><id>1</id><name>User1</name></entity><entity><id>2</id><name>User2</name></entity><entity><id>3</id><name>User3</name></entity><entity><id>4</id><name>User4</name></entity><entity><id>5</id><name>User5</name></entity>

      Mapper was expected to get value as

      Value 1 - <entity><id>1</id><name>User1</name>
      Value 2 - <entity><id>2</id><name>User2</name>
      Value 3 - <entity><id>3</id><name>User3</name>
      Value 4 - <entity><id>4</id><name>User4</name>
      Value 5 - <entity><id>5</id><name>User5</name>

      According to this bug Mapper gets value

      Value 1 - entity><id>1</id><name>User1</name>
      Value 2 - <entity>id>2</id><name>User2</name>
      Value 3 - <entity><id>3id><name>User3</name>
      Value 4 - <entity><id>4</id><name>User4name>
      Value 5 - <entity><id>5</id><name>User5</name>

      The pattern shown above need not occur for value 1,2,3 necessarily. The bug occurs at some random positions in the map input.

      1. HADOOP-8655 (2).patch
        11 kB
        Gelesh
      2. HADOOP-8655.patch
        11 kB
        Gelesh
      3. HADOOP-8655.patch
        10 kB
        Gelesh
      4. HADOOP-8655.patch
        10 kB
        Gelesh
      5. MAPREDUCE-4519.patch
        4 kB
        Meria Joseph

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1175 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1175/)
        HADOOP-8655. Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1175 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1175/ ) HADOOP-8655 . Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1144 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1144/)
        HADOOP-8655. Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1144 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1144/ ) HADOOP-8655 . Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2655 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2655/)
        HADOOP-8655. Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2655 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2655/ ) HADOOP-8655 . Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2691 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2691/)
        HADOOP-8655. Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2691 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2691/ ) HADOOP-8655 . Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2627 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2627/)
        HADOOP-8655. Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2627 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2627/ ) HADOOP-8655 . Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376592 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
        Hide
        Robert Joseph Evans added a comment -

        Thanks for the patch Gelesh +1.

        I checked this into trunk and branch-2.

        Show
        Robert Joseph Evans added a comment - Thanks for the patch Gelesh +1. I checked this into trunk and branch-2.
        Hide
        Arun A K added a comment -

        Gelesh :I had tried out the patch that you have posted herein. That really solves my problem. Thanks a lot for the patch. Is the patch that you re uploaded same as before? Do I need to apply this new patch?

        Show
        Arun A K added a comment - Gelesh :I had tried out the patch that you have posted herein. That really solves my problem. Thanks a lot for the patch. Is the patch that you re uploaded same as before? Do I need to apply this new patch?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542076/HADOOP-8655%20%282%29.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.ha.TestZKFailoverController

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1348//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1348//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542076/HADOOP-8655%20%282%29.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1348//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1348//console This message is automatically generated.
        Hide
        Gelesh added a comment -

        Since Hadoop_QA automated testing has not acted upon the previous patch, re uploading the same

        Show
        Gelesh added a comment - Since Hadoop_QA automated testing has not acted upon the previous patch, re uploading the same
        Hide
        Gelesh added a comment -

        Thank you Robert Joseph Evans,
        This patch is updated as per your comments

        Show
        Gelesh added a comment - Thank you Robert Joseph Evans, This patch is updated as per your comments
        Hide
        Robert Joseph Evans added a comment -

        Gelesh,

        The new patch looks better, but I still have a few comments.

        1. Please make sure you follow the style guide. It should follow Sun's code conventions except indentation is 2 spaces, not 4. There are still tabs everywhere throughout the code and there are many lines that go over 80 characters in length. Comments are included in the 80 character limit.
        2. In the test getTestData method is only called once, and is very specific to the single test method. I would prefer to see it inlined in testCustomDeliminator.
        3. I appreciate that you want to explain what is happening in your code, but I don't think you need quite so many comments. For example you don't need to reference HADOOP-8654. There should be test cases added with HADOOP-8654 to validate that there were no regression.
        Show
        Robert Joseph Evans added a comment - Gelesh, The new patch looks better, but I still have a few comments. Please make sure you follow the style guide. It should follow Sun's code conventions except indentation is 2 spaces, not 4. There are still tabs everywhere throughout the code and there are many lines that go over 80 characters in length. Comments are included in the 80 character limit. In the test getTestData method is only called once, and is very specific to the single test method. I would prefer to see it inlined in testCustomDeliminator. I appreciate that you want to explain what is happening in your code, but I don't think you need quite so many comments. For example you don't need to reference HADOOP-8654 . There should be test cases added with HADOOP-8654 to validate that there were no regression.
        Hide
        Gelesh added a comment -

        Thanks Robert Joseph Evans & Jason Lowe , for providing the info,
        If I am not wrong, ZKFailoverController itself has a problem , and that is being reflected here.
        If so, I hope this could be closed,
        Lets listen from Arun AK, as well,
        Hope his data sets would respond positevely.

        Show
        Gelesh added a comment - Thanks Robert Joseph Evans & Jason Lowe , for providing the info, If I am not wrong, ZKFailoverController itself has a problem , and that is being reflected here. If so, I hope this could be closed, Lets listen from Arun AK, as well, Hope his data sets would respond positevely.
        Hide
        Jason Lowe added a comment -

        The TestZKFailoverController failure is unrelated, see HADOOP-8591.

        Show
        Jason Lowe added a comment - The TestZKFailoverController failure is unrelated, see HADOOP-8591 .
        Hide
        Gelesh added a comment -

        Could any body clarify about
        org.apache.hadoop.ha.TestZKFailoverController Unit Test

        Show
        Gelesh added a comment - Could any body clarify about org.apache.hadoop.ha.TestZKFailoverController Unit Test
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12541726/HADOOP-8655.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.ha.TestZKFailoverController

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1339//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1339//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541726/HADOOP-8655.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1339//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1339//console This message is automatically generated.
        Hide
        Gelesh added a comment -

        Revised the patch as per Robert Joseph Evans comments

        Show
        Gelesh added a comment - Revised the patch as per Robert Joseph Evans comments
        Hide
        Robert Joseph Evans added a comment -

        The code looks good, but it looks like you put TestLineReader.java under the main directory when it should be under the test directory. It will not compile under main. I also haven't had a chance to look at it in depth.

        Show
        Robert Joseph Evans added a comment - The code looks good, but it looks like you put TestLineReader.java under the main directory when it should be under the test directory. It will not compile under main. I also haven't had a chance to look at it in depth.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12541232/HADOOP-8655.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1320//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541232/HADOOP-8655.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1320//console This message is automatically generated.
        Hide
        Gelesh added a comment -

        Patch is commited aganist 04ba22681a494bf718dff7926e783c75bf64c2c7 taking care of HADOOP-8654,

        Show
        Gelesh added a comment - Patch is commited aganist 04ba22681a494bf718dff7926e783c75bf64c2c7 taking care of HADOOP-8654 ,
        Hide
        Jason Lowe added a comment -

        Comments from a quick perusal of the patch:

        • Please post patches with appropriate names, this is HADOOP-8655 but the patch name implies it's for HADOOP-8654.
        • Patch needs to be updated to trunk. HADOOP-8654 has been committed since this patch was posted.
        • Patch contains tabs, please convert to spaces.
        • Why were the InterfaceAudience and InterfaceStability decorators removed?
        Show
        Jason Lowe added a comment - Comments from a quick perusal of the patch: Please post patches with appropriate names, this is HADOOP-8655 but the patch name implies it's for HADOOP-8654 . Patch needs to be updated to trunk. HADOOP-8654 has been committed since this patch was posted. Patch contains tabs, please convert to spaces. Why were the InterfaceAudience and InterfaceStability decorators removed?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12541222/HADOOP-8654.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1318//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541222/HADOOP-8654.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1318//console This message is automatically generated.
        Hide
        Gelesh added a comment -

        Patch with test case. This patch holds good with the test case of HADOOP-8654.

        Show
        Gelesh added a comment - Patch with test case. This patch holds good with the test case of HADOOP-8654 .
        Hide
        Meria Joseph added a comment -

        The issue occurs when the buffer that reads the input file content, at a particular instance, ends with a character or character sequence that matches the head of the record delimiter.

        For example, in the above case, while reading the file, the buffer's end bytes at an instance might be as follows,

        ........</name></entity><entity><id>3</

        causing it to skip the last two characters considering it as a part of the delimiter </entity>.

        The default buffer size is 4096 bytes.Hence the input should be more than 4096 bytes and the last bytes of the buffer should match the head of the delimiter...Please guide how to create test case for the patch..

        Show
        Meria Joseph added a comment - The issue occurs when the buffer that reads the input file content, at a particular instance, ends with a character or character sequence that matches the head of the record delimiter. For example, in the above case, while reading the file, the buffer's end bytes at an instance might be as follows, ........</name></entity><entity><id>3</ causing it to skip the last two characters considering it as a part of the delimiter </entity>. The default buffer size is 4096 bytes.Hence the input should be more than 4096 bytes and the last bytes of the buffer should match the head of the delimiter...Please guide how to create test case for the patch..
        Hide
        Jason Lowe added a comment -

        Finding the right JIRA project is a straightforward mapping from the top-level projects in the code base:

        • Anything under hadoop-common-project maps to Hadoop Common
        • Anything under hadoop-hdfs-project maps to Hadoop HDFS
        • Anything under hadoop-mapreduce-project maps to Hadoop Map/Reduce
        Show
        Jason Lowe added a comment - Finding the right JIRA project is a straightforward mapping from the top-level projects in the code base: Anything under hadoop-common-project maps to Hadoop Common Anything under hadoop-hdfs-project maps to Hadoop HDFS Anything under hadoop-mapreduce-project maps to Hadoop Map/Reduce
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12539291/MAPREDUCE-4519.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1253//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1253//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539291/MAPREDUCE-4519.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1253//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1253//console This message is automatically generated.
        Hide
        Gelesh added a comment -

        Thank you Meria for including my patch as well.
        Thats Jason, for have a closer look, and merging it.
        Please guide about howt to categorize , the bug
        As this was a issue faced in MAP REDUCE
        And was supposed to be raised in HADOOP

        Show
        Gelesh added a comment - Thank you Meria for including my patch as well. Thats Jason, for have a closer look, and merging it. Please guide about howt to categorize , the bug As this was a issue faced in MAP REDUCE And was supposed to be raised in HADOOP
        Hide
        Jason Lowe added a comment -

        As with MAPREDUCE-4512, I moved this to project Hadoop Common since that's where the patch needs to be applied.

        In the future, please don't set the Reviewed flag unless the patch has been reviewed and approved by someone in the community. I see no record of that occurring, so I've cleared that flag. Also the Fix versions field is intended to mark where the patch has been integrated, so please don't set this field. If you'd like to indicate what versions you'd like to have the patch committed to, use the Target Versions field instead.

        Show
        Jason Lowe added a comment - As with MAPREDUCE-4512 , I moved this to project Hadoop Common since that's where the patch needs to be applied. In the future, please don't set the Reviewed flag unless the patch has been reviewed and approved by someone in the community. I see no record of that occurring, so I've cleared that flag. Also the Fix versions field is intended to mark where the patch has been integrated, so please don't set this field. If you'd like to indicate what versions you'd like to have the patch committed to, use the Target Versions field instead.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12539291/MAPREDUCE-4519.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2709//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2709//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539291/MAPREDUCE-4519.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2709//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2709//console This message is automatically generated.
        Hide
        Meria Joseph added a comment -

        A few lines of change in LineReader, also incorporaed the MAPREDUCE-4512 patch

        Show
        Meria Joseph added a comment - A few lines of change in LineReader, also incorporaed the MAPREDUCE-4512 patch
        Hide
        Meria Joseph added a comment -

        A few lines of change in LineReader, also incorporaed the MAPREDUCE-4512 patch

        Show
        Meria Joseph added a comment - A few lines of change in LineReader, also incorporaed the MAPREDUCE-4512 patch
        Hide
        Gelesh added a comment -

        I have found a similar Bug And a fix, MAPREDUCE-4512. Please reffer the patch, and kindly encorporate the same.
        While fixing I too have encounted such a senario, I think this occur at the end of the buffer which would capture 4096 Charactors.
        My understanding is the ending and begining of next buffer can and the delimiter indexses are not properly handled.
        This is resulting in some or the other bugs.

        Tried solving , but the fix resulted in some new bugs. The once all the senario is caught we can ensure a posible fix.

        Show
        Gelesh added a comment - I have found a similar Bug And a fix, MAPREDUCE-4512 . Please reffer the patch, and kindly encorporate the same. While fixing I too have encounted such a senario, I think this occur at the end of the buffer which would capture 4096 Charactors. My understanding is the ending and begining of next buffer can and the delimiter indexses are not properly handled. This is resulting in some or the other bugs. Tried solving , but the fix resulted in some new bugs. The once all the senario is caught we can ensure a posible fix.

          People

          • Assignee:
            Unassigned
            Reporter:
            Arun A K
          • Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 168h
              168h
              Remaining:
              Remaining Estimate - 168h
              168h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development