Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-96

HDFS does not support blocks greater than 2GB

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS currently does not support blocks greater than 2GB in size.

      1. HDFS-96.2.patch
        10 kB
        Patrick Kling
      2. HDFS-96.patch
        10 kB
        Patrick Kling
      3. hdfslargeblkcrash.tar.gz
        19 kB
        Vinay Setty
      4. largeBlockSize1.txt
        7 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          It has been suggested to merge this to branch-1. Current patches do not apply and will require porting. Opened HDFS-3617 to track this work.

          Show
          Matt Foley added a comment - It has been suggested to merge this to branch-1. Current patches do not apply and will require porting. Opened HDFS-3617 to track this work.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #406 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/406/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #406 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/406/ )
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Patrick.

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Patrick.
          Hide
          Patrick Kling added a comment -

          HDFS-96.2.patch QA results:

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 3 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          [exec]
          [exec] +1 system tests framework. The patch passed system tests framework compile.

          Show
          Patrick Kling added a comment - HDFS-96 .2.patch QA results: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system tests framework. The patch passed system tests framework compile.
          Hide
          Patrick Kling added a comment -

          revised version of the patch that avoids deprecation warning when compiling test case

          Show
          Patrick Kling added a comment - revised version of the patch that avoids deprecation warning when compiling test case
          Hide
          Patrick Kling added a comment -

          This patch fixes an integer overflow problem that causes a failure with block sizes of more than 2GB.

          The added test in org.apache.hadoop.hdfs.TestLargeBlock verifies that we can create, write to and correctly read from a file when the block size is > 2 GB.

          The following test cases fail both on the current version of trunk before applying this patch and after applying this patch (i.e., no new test failures were introduced by this patch):

          [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED
          [junit] Test org.apache.hadoop.hdfs.TestFileStatus FAILED
          [junit] Test org.apache.hadoop.hdfs.TestHDFSServerPorts FAILED
          [junit] Test org.apache.hadoop.hdfs.TestHDFSTrash FAILED (timeout)
          [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBackupNode FAILED
          [junit] Test org.apache.hadoop.fs.TestHDFSFileContextMainOperations FAILED
          [junit] Test org.apache.hadoop.hdfs.TestFileConcurrentReader FAILED (timeout)
          [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockReport FAILED
          [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS FAILED
          [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED
          [junit] Test org.apache.hadoop.hdfs.TestFiHFlush FAILED

          Show
          Patrick Kling added a comment - This patch fixes an integer overflow problem that causes a failure with block sizes of more than 2GB. The added test in org.apache.hadoop.hdfs.TestLargeBlock verifies that we can create, write to and correctly read from a file when the block size is > 2 GB. The following test cases fail both on the current version of trunk before applying this patch and after applying this patch (i.e., no new test failures were introduced by this patch): [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED [junit] Test org.apache.hadoop.hdfs.TestFileStatus FAILED [junit] Test org.apache.hadoop.hdfs.TestHDFSServerPorts FAILED [junit] Test org.apache.hadoop.hdfs.TestHDFSTrash FAILED (timeout) [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBackupNode FAILED [junit] Test org.apache.hadoop.fs.TestHDFSFileContextMainOperations FAILED [junit] Test org.apache.hadoop.hdfs.TestFileConcurrentReader FAILED (timeout) [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockReport FAILED [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS FAILED [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED [junit] Test org.apache.hadoop.hdfs.TestFiHFlush FAILED
          Hide
          Vinay Setty added a comment -

          Attaching detailed description of problem faced and also attaching the logs after all the fixes we did mentioned in the description.

          Show
          Vinay Setty added a comment - Attaching detailed description of problem faced and also attaching the logs after all the fixes we did mentioned in the description.
          Hide
          dhruba borthakur added a comment -

          I have not yet found all the problems yet. This is mostly a unit-test to catch this problem. The first of the fixes is a integer-to-long conversion problem:

          • int len = Math.min((int) (endOffset - offset),
          • bytesPerChecksum*maxChunks);
            + int len = (int)Math.min(endOffset - offset,
            + (long)(bytesPerChecksum*maxChunks));
          Show
          dhruba borthakur added a comment - I have not yet found all the problems yet. This is mostly a unit-test to catch this problem. The first of the fixes is a integer-to-long conversion problem: int len = Math.min((int) (endOffset - offset), bytesPerChecksum*maxChunks); + int len = (int)Math.min(endOffset - offset, + (long)(bytesPerChecksum*maxChunks));
          Hide
          Konstantin Shvachko added a comment -

          Could you please clarify what is the problem with large blocks and how you fix it.

          Show
          Konstantin Shvachko added a comment - Could you please clarify what is the problem with large blocks and how you fix it.
          Hide
          dhruba borthakur added a comment -

          Here is a unit test that demonstrates that bug. Also, a simple fix (this might not be all that is required).

          Show
          dhruba borthakur added a comment - Here is a unit test that demonstrates that bug. Also, a simple fix (this might not be all that is required).

            People

            • Assignee:
              Patrick Kling
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development