Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11096 Support rolling upgrade between 2.x and 3.x
  3. HDFS-12151

Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha4
    • 3.0.0-beta1
    • rolling upgrades
    • None

    Description

      Trying to write to a Hadoop 3 DataNode with a Hadoop 2 client currently fails. On the client side it looks like this:

          17/07/14 13:31:58 INFO hdfs.DFSClient: Exception in createBlockOutputStream
          java.io.EOFException: Premature EOF: no length prefix available
                  at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280)
                  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
                  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237)
                  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)

      But on the DataNode side there's an ArrayOutOfBoundsException because there aren't any targetStorageIds:

          java.lang.ArrayIndexOutOfBoundsException: 0
                  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:815)
                  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
                  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
                  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
                  at java.lang.Thread.run(Thread.java:745)

      Attachments

        1. HDFS-12151.007.patch
          10 kB
          Sean Mackrory
        2. HDFS-12151.006.patch
          10 kB
          Sean Mackrory
        3. HDFS-12151.005.patch
          11 kB
          Sean Mackrory
        4. HDFS-12151.004.patch
          11 kB
          Sean Mackrory
        5. HDFS-12151.003.patch
          9 kB
          Sean Mackrory
        6. HDFS-12151.002.patch
          9 kB
          Sean Mackrory
        7. HDFS-12151.001.patch
          2 kB
          Sean Mackrory

        Issue Links

          Activity

            People

              mackrorysd Sean Mackrory
              mackrorysd Sean Mackrory
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: