Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1527

SocketOutputStream.transferToFully fails for blocks >= 2GB on 32 bit JVM

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: datanode
    • Labels:
      None
    • Environment:

      32 bit JVM

    • Hadoop Flags:
      Reviewed

      Description

      On 32 bit JVM, SocketOutputStream.transferToFully() fails if the block size is >= 2GB. We should fall back to a normal transfer in this case.

      2010-12-02 19:04:23,490 ERROR datanode.DataNode (BlockSender.java:sendChunks(399)) - BlockSender.sendChunks() exception: java.io.IOException: Value too large
       for defined data type
              at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
              at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:418)
              at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:519)
              at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:204)
              at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:386)
              at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:475)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opReadBlock(DataXceiver.java:196)
              at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opReadBlock(DataTransferProtocol.java:356)
              at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:328)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:130)
              at java.lang.Thread.run(Thread.java:619)
      
      1. HDFS-1527.2.patch
        3 kB
        Patrick Kling
      2. HDFS-1527.patch
        2 kB
        Patrick Kling

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Hide
          Konstantin Boudnik added a comment -

          I have just committed this. Thanks Patrick.

          Show
          Konstantin Boudnik added a comment - I have just committed this. Thanks Patrick.
          Hide
          Konstantin Boudnik added a comment -

          HDFS-1523 was committed to branch-0.22 and trunk. So, I believe this has to be applied to both as well.

          Show
          Konstantin Boudnik added a comment - HDFS-1523 was committed to branch-0.22 and trunk. So, I believe this has to be applied to both as well.
          Hide
          Konstantin Boudnik added a comment -

          Ok, that makes sense.

          +1 patch looks good. I will commit it shortly unless someone has an objection.

          Show
          Konstantin Boudnik added a comment - Ok, that makes sense. +1 patch looks good. I will commit it shortly unless someone has an objection.
          Hide
          Patrick Kling added a comment -

          All of the tests mentioned above fail on a clean check-out of trunk on my machine. Interestingly, TestBlockReport succeeds on my machine (but not on hudson).

          Show
          Patrick Kling added a comment - All of the tests mentioned above fail on a clean check-out of trunk on my machine. Interestingly, TestBlockReport succeeds on my machine (but not on hudson).
          Hide
          Konstantin Boudnik added a comment -

          Current trunk build #513 has only 8 test cases failing from 5 tests. Some of the failures above are different from the trunk ones.

          Show
          Konstantin Boudnik added a comment - Current trunk build #513 has only 8 test cases failing from 5 tests. Some of the failures above are different from the trunk ones.
          Hide
          Patrick Kling added a comment -

          I have now run the complete set of HDFS tests. The only tests that fail are the ones that also fail on a clean trunk:

              [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED
              [junit] Test org.apache.hadoop.hdfs.TestHDFSServerPorts FAILED
              [junit] Test org.apache.hadoop.hdfs.TestHDFSTrash FAILED (timeout)
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBackupNode FAILED
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestStorageRestore FAILED
              [junit] Test org.apache.hadoop.hdfs.TestFileConcurrentReader FAILED (timeout)
              [junit] Test org.apache.hadoop.hdfs.server.balancer.TestBalancer FAILED
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS FAILED
              [junit] Test org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete FAILED (timeout)
              [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED
          
          Show
          Patrick Kling added a comment - I have now run the complete set of HDFS tests. The only tests that fail are the ones that also fail on a clean trunk: [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED [junit] Test org.apache.hadoop.hdfs.TestHDFSServerPorts FAILED [junit] Test org.apache.hadoop.hdfs.TestHDFSTrash FAILED (timeout) [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBackupNode FAILED [junit] Test org.apache.hadoop.hdfs.server.namenode.TestStorageRestore FAILED [junit] Test org.apache.hadoop.hdfs.TestFileConcurrentReader FAILED (timeout) [junit] Test org.apache.hadoop.hdfs.server.balancer.TestBalancer FAILED [junit] Test org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS FAILED [junit] Test org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete FAILED (timeout) [junit] Test org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery FAILED
          Hide
          Patrick Kling added a comment -

          Removed the unused import. run-commit-test and run-smoke-test yield the same results as on a clean trunk. I'll post the full HDFS test results once they have completed.

          Show
          Patrick Kling added a comment - Removed the unused import. run-commit-test and run-smoke-test yield the same results as on a clean trunk. I'll post the full HDFS test results once they have completed.
          Hide
          Konstantin Boudnik added a comment -

          This line needs to go to for it become unused import

          import static org.junit.Assume.assumeTrue;
          

          Rest looks ok to me. Have you ran the rest of HDFS tests with this patch?

          Show
          Konstantin Boudnik added a comment - This line needs to go to for it become unused import import static org.junit.Assume.assumeTrue; Rest looks ok to me. Have you ran the rest of HDFS tests with this patch?
          Hide
          Patrick Kling added a comment -

          This patch uses a regular transfer instead of transferTo if we are on a 32bit JVM and the block size is >= Integer.MAX_INT. I also re-enables TestLargeBlock on 32bit.

          ant test-patch output:

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     +1 system test framework.  The patch passed system test framework compile.
          
          Show
          Patrick Kling added a comment - This patch uses a regular transfer instead of transferTo if we are on a 32bit JVM and the block size is >= Integer.MAX_INT. I also re-enables TestLargeBlock on 32bit. ant test-patch output: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile.

            People

            • Assignee:
              Patrick Kling
              Reporter:
              Patrick Kling
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development