Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4888

NLineInputFormat drops data in 1.1 and beyond

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.2
    • Component/s: mrv1
    • Labels:
      None

      Description

      When trying to root cause why MAPREDUCE-4782 did not cause us issues on 1.0.2, I found out that HADOOP-7823 introduced essentially the exact same error into org.apache.hadoop.mapred.lib.NLineInputFormat.

      In 1.X org.apache.hadoop.mapred.lib.NLineInputFormat and org.apache.hadoop.mapreduce.lib.input.NLineInputFormat are separate implementations. The latter had an off by one error in it until MAPREDUCE-4782 fixed it. The former had no error in it until HADOOP-7823 introduced it in 1.1 and MAPREDUCE-375 combined the implementations together but picked the implementation with the off by one error in 0.21.

      I will attach a patch that exposes the error.

      1. test.patch
        3 kB
        Robert Joseph Evans
      2. MAPREDUCE-4888-20121219.txt
        5 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon successful release of 1.1.2.

          Show
          Matt Foley added a comment - Closed upon successful release of 1.1.2.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Vinod and Bobby!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Vinod and Bobby!
          Hide
          Robert Joseph Evans added a comment -

          But that is very minor and I would not want it to stop this from going in.

          Show
          Robert Joseph Evans added a comment - But that is very minor and I would not want it to stop this from going in.
          Hide
          Robert Joseph Evans added a comment - - edited

          I am +1 too. My only comment is you may want to delete out my debugging System.out.println calls in the test.

          Show
          Robert Joseph Evans added a comment - - edited I am +1 too. My only comment is you may want to delete out my debugging System.out.println calls in the test.
          Hide
          Chris Douglas added a comment -

          +1

          Show
          Chris Douglas added a comment - +1
          Hide
          Vinod Kumar Vavilapalli added a comment -

          test-patch results:

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     -1 findbugs.  The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings.
               [exec] 
               [exec] 
               [exec] 
               [exec] 
               [exec] ======================================================================
               [exec] ======================================================================
               [exec]     Finished build.
               [exec] ======================================================================
               [exec] ======================================================================
          

          None of the warnings are new.

          Running tests.

          Show
          Vinod Kumar Vavilapalli added a comment - test-patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== None of the warnings are new. Running tests.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Bobby, have time for a quick review? Tx.

          Show
          Vinod Kumar Vavilapalli added a comment - Bobby, have time for a quick review? Tx.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Copying changes over from MAPREDUCE-4782. The test passes after the changes.

          Show
          Vinod Kumar Vavilapalli added a comment - Copying changes over from MAPREDUCE-4782 . The test passes after the changes.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Bobby, taking this over. Thanks for your analysis and the test-case, it made my job easier.

          Show
          Vinod Kumar Vavilapalli added a comment - Bobby, taking this over. Thanks for your analysis and the test-case, it made my job easier.
          Hide
          Robert Joseph Evans added a comment -

          I really don't have time to work on this right now, if someone else wants to take a look at it that would be great. It probably will just involve undoing the changes to NLineInputFormat introduced in HADOOP-7823.

          Show
          Robert Joseph Evans added a comment - I really don't have time to work on this right now, if someone else wants to take a look at it that would be great. It probably will just involve undoing the changes to NLineInputFormat introduced in HADOOP-7823 .
          Hide
          Robert Joseph Evans added a comment -

          This patch exposes the issue. It is mostly just a port of the test changes that were made as a part of MAPREDUCE-4782.

          Show
          Robert Joseph Evans added a comment - This patch exposes the issue. It is mostly just a port of the test changes that were made as a part of MAPREDUCE-4782 .

            People

            • Assignee:
              Vinod Kumar Vavilapalli
              Reporter:
              Robert Joseph Evans
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development