Hadoop Common
  1. Hadoop Common
  2. HADOOP-9544

backport UTF8 encoding fixes to branch-1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.2.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The trunk code has received numerous bug fixes related to UTF8 encoding. I recently observed a branch-1-based cluster fail to load its fsimage due to these bugs. I've confirmed that the bug fixes existing on trunk will resolve this, so I'd like to backport to branch-1.

      1. HDFS-4795-branch-1.2.patch
        16 kB
        Chris Nauroth
      2. HDFS-4795-branch-1.1.patch
        35 kB
        Chris Nauroth

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Hide
          Chris Nauroth added a comment -

          Thanks very much for the review and commit.

          Just to document it, my full test run came back successful. There was just 1 failure in TestBalancerWithNodeGroup, which is known to be a flaky test.

          Show
          Chris Nauroth added a comment - Thanks very much for the review and commit. Just to document it, my full test run came back successful. There was just 1 failure in TestBalancerWithNodeGroup , which is known to be a flaky test.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Chris!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Chris!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I also have run test-patch. I consistently see 19 Findbugs warnings in my machine. So that I believe they are not related to the patch.

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 5 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     -1 findbugs.  The patch appears to introduce 19 new Findbugs (version 1.3.9) warnings.
          
          Show
          Tsz Wo Nicholas Sze added a comment - I also have run test-patch. I consistently see 19 Findbugs warnings in my machine. So that I believe they are not related to the patch. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 5 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 19 new Findbugs (version 1.3.9) warnings.
          Hide
          Chris Nauroth added a comment -

          Thanks, Nicholas. I'll report back when I have the test results. Here is the test-patch output. Regarding findbugs, it appears that the filterBugs command in test-patch.sh is not actually filtering out anything. I've seen mention of other people having this problem on the mailing lists. It's unrelated to this patch.

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 5 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     -1 findbugs.  The patch appears to introduce 427 new Findbugs (version 1.3.9) warnings.
          
          Show
          Chris Nauroth added a comment - Thanks, Nicholas. I'll report back when I have the test results. Here is the test-patch output. Regarding findbugs, it appears that the filterBugs command in test-patch.sh is not actually filtering out anything. I've seen mention of other people having this problem on the mailing lists. It's unrelated to this patch. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 5 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 427 new Findbugs (version 1.3.9) warnings.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Please run all the unit tests and post the results.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good. Please run all the unit tests and post the results.
          Hide
          Chris Nauroth added a comment -

          I'm uploading version 2 of the patch. I realized that the Guava dependency caused by GenericTestUtils could be problematic. This version of the patch does not introduce a Guava dependency, and instead uses a stripped down version of GenericTestUtils that contains just enough to support the new tests in TestUTF8.

          Show
          Chris Nauroth added a comment - I'm uploading version 2 of the patch. I realized that the Guava dependency caused by GenericTestUtils could be problematic. This version of the patch does not introduce a Guava dependency, and instead uses a stripped down version of GenericTestUtils that contains just enough to support the new tests in TestUTF8 .
          Hide
          Chris Nauroth added a comment -

          I'm attaching the backport patch. The prior jiras included in this backport are: HADOOP-6522, HADOOP-7547 (partially), HADOOP-9103, HDFS-3844 (partially), and HDFS-4282.

          Porting in the new tests also required including several dependency classes that previously didn't exist in branch-1: Time, GenericTestUtils and TimedOutTestsListener.

          One of the changes in trunk had been in FSImageSerialization, which doesn't exist in branch-1. I made the equivalent change in FSImage.

          Show
          Chris Nauroth added a comment - I'm attaching the backport patch. The prior jiras included in this backport are: HADOOP-6522 , HADOOP-7547 (partially), HADOOP-9103 , HDFS-3844 (partially), and HDFS-4282 . Porting in the new tests also required including several dependency classes that previously didn't exist in branch-1: Time , GenericTestUtils and TimedOutTestsListener . One of the changes in trunk had been in FSImageSerialization , which doesn't exist in branch-1. I made the equivalent change in FSImage .

            People

            • Assignee:
              Chris Nauroth
              Reporter:
              Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development