Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12840

Creating a file with non-default EC policy in a EC zone is not correctly serialized in the editlog

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Add ErasureCodingPolicyId to each OP_ADD edit log op.

      Description

      When create a replicated file in an existing EC zone, the edit logs does not differentiate it from an EC file. When FSEditLogLoader to replay edits, this file is treated as EC file, as a results, it crashes the NN because the blocks of this file are replicated, which does not match with INode.

      ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception on operation AddBlockOp [path=/system/balancer.id, penultimateBlock=NULL, lastBlock=blk_1073743259_2455, RpcClientId=, RpcCallId=-2]
      java.lang.IllegalArgumentException: reportedBlock is not striped
      	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStriped.addStorage(BlockInfoStriped.java:118)
      	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.addBlock(DatanodeStorageInfo.java:256)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlock(BlockManager.java:3141)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlockUnderConstruction(BlockManager.java:3068)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processAndHandleReportedBlock(BlockManager.java:3864)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processQueuedMessages(BlockManager.java:2916)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processQueuedMessagesForBlock(BlockManager.java:2903)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.addNewBlock(FSEditLogLoader.java:1069)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:532)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
      	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:293)
      	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:427)
      	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:380)
      	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:397)
      

        Attachments

        1. HDFS-12840.reprod.patch
          2 kB
          Lei (Eddy) Xu
        2. HDFS-12840.05.patch
          57 kB
          Lei (Eddy) Xu
        3. HDFS-12840.04.patch
          56 kB
          Lei (Eddy) Xu
        4. HDFS-12840.03.patch
          56 kB
          Lei (Eddy) Xu
        5. HDFS-12840.02.patch
          57 kB
          Lei (Eddy) Xu
        6. HDFS-12840.01.patch
          80 kB
          Lei (Eddy) Xu
        7. HDFS-12840.00.patch
          14 kB
          Lei (Eddy) Xu
        8. editsStored.05
          8 kB
          Lei (Eddy) Xu
        9. editsStored.03
          8 kB
          Lei (Eddy) Xu
        10. editsStored
          8 kB
          Lei (Eddy) Xu
        11. editsStored
          8 kB
          Lei (Eddy) Xu

          Issue Links

            Activity

              People

              • Assignee:
                eddyxu Lei (Eddy) Xu
                Reporter:
                eddyxu Lei (Eddy) Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: