Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12840

Creating a file with non-default EC policy in a EC zone is not correctly serialized in the editlog

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed
    • Add ErasureCodingPolicyId to each OP_ADD edit log op.

    Description

      When create a replicated file in an existing EC zone, the edit logs does not differentiate it from an EC file. When FSEditLogLoader to replay edits, this file is treated as EC file, as a results, it crashes the NN because the blocks of this file are replicated, which does not match with INode.

      ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception on operation AddBlockOp [path=/system/balancer.id, penultimateBlock=NULL, lastBlock=blk_1073743259_2455, RpcClientId=, RpcCallId=-2]
      java.lang.IllegalArgumentException: reportedBlock is not striped
      	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStriped.addStorage(BlockInfoStriped.java:118)
      	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.addBlock(DatanodeStorageInfo.java:256)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlock(BlockManager.java:3141)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlockUnderConstruction(BlockManager.java:3068)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processAndHandleReportedBlock(BlockManager.java:3864)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processQueuedMessages(BlockManager.java:2916)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processQueuedMessagesForBlock(BlockManager.java:2903)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.addNewBlock(FSEditLogLoader.java:1069)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:532)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
      	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:293)
      	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:427)
      	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:380)
      	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:397)
      

      Attachments

        1. HDFS-12840.reprod.patch
          2 kB
          Lei (Eddy) Xu
        2. HDFS-12840.05.patch
          57 kB
          Lei (Eddy) Xu
        3. HDFS-12840.04.patch
          56 kB
          Lei (Eddy) Xu
        4. HDFS-12840.03.patch
          56 kB
          Lei (Eddy) Xu
        5. HDFS-12840.02.patch
          57 kB
          Lei (Eddy) Xu
        6. HDFS-12840.01.patch
          80 kB
          Lei (Eddy) Xu
        7. HDFS-12840.00.patch
          14 kB
          Lei (Eddy) Xu
        8. editsStored.05
          8 kB
          Lei (Eddy) Xu
        9. editsStored.03
          8 kB
          Lei (Eddy) Xu
        10. editsStored
          8 kB
          Lei (Eddy) Xu
        11. editsStored
          8 kB
          Lei (Eddy) Xu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            eddyxu Lei (Eddy) Xu Assign to me
            eddyxu Lei (Eddy) Xu
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment