Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1527

SocketOutputStream.transferToFully fails for blocks >= 2GB on 32 bit JVM

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.22.0
    • datanode
    • None
    • 32 bit JVM

    • Reviewed

    Description

      On 32 bit JVM, SocketOutputStream.transferToFully() fails if the block size is >= 2GB. We should fall back to a normal transfer in this case.

      2010-12-02 19:04:23,490 ERROR datanode.DataNode (BlockSender.java:sendChunks(399)) - BlockSender.sendChunks() exception: java.io.IOException: Value too large
       for defined data type
              at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
              at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:418)
              at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:519)
              at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:204)
              at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:386)
              at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:475)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opReadBlock(DataXceiver.java:196)
              at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opReadBlock(DataTransferProtocol.java:356)
              at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:328)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:130)
              at java.lang.Thread.run(Thread.java:619)
      

      Attachments

        1. HDFS-1527.2.patch
          3 kB
          Patrick Kling
        2. HDFS-1527.patch
          2 kB
          Patrick Kling

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            pkling Patrick Kling
            pkling Patrick Kling
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment