Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11096 Support rolling upgrade between 2.x and 3.x
  3. HDFS-12151

Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: 3.0.0-beta1
    • Component/s: rolling upgrades
    • Labels:
      None

      Description

      Trying to write to a Hadoop 3 DataNode with a Hadoop 2 client currently fails. On the client side it looks like this:

          17/07/14 13:31:58 INFO hdfs.DFSClient: Exception in createBlockOutputStream
          java.io.EOFException: Premature EOF: no length prefix available
                  at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280)
                  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
                  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237)
                  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)

      But on the DataNode side there's an ArrayOutOfBoundsException because there aren't any targetStorageIds:

          java.lang.ArrayIndexOutOfBoundsException: 0
                  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:815)
                  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
                  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
                  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
                  at java.lang.Thread.run(Thread.java:745)

        Attachments

        1. HDFS-12151.001.patch
          2 kB
          Sean Mackrory
        2. HDFS-12151.002.patch
          9 kB
          Sean Mackrory
        3. HDFS-12151.003.patch
          9 kB
          Sean Mackrory
        4. HDFS-12151.004.patch
          11 kB
          Sean Mackrory
        5. HDFS-12151.005.patch
          11 kB
          Sean Mackrory
        6. HDFS-12151.006.patch
          10 kB
          Sean Mackrory
        7. HDFS-12151.007.patch
          10 kB
          Sean Mackrory

          Issue Links

            Activity

              People

              • Assignee:
                mackrorysd Sean Mackrory
                Reporter:
                mackrorysd Sean Mackrory
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: