Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • HDFS-7285
    • namenode
    • None
    • Reviewed

    Description

      Since random block IDs were supported by some early version of HDFS, the block ID reserved for EC blocks could be already used by some existing blocks in a cluster. During NameNode startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap lookup.

      Attachments

        1. HDFS-7994_001.patch
          9 kB
          Hui Zheng
        2. HDFS-7994_002.patch
          19 kB
          Hui Zheng

        Activity

          huizane Hui Zheng added a comment -

          Hi Nicholas
          I would like to work on this issue.

          huizane Hui Zheng added a comment - Hi Nicholas I would like to work on this issue.
          szetszwo Tsz-wo Sze added a comment -

          That's great! Thanks.

          szetszwo Tsz-wo Sze added a comment - That's great! Thanks.
          huizane Hui Zheng added a comment -

          Hi szetszwo
          I want to confirm my understanding about this issue.Is it correct?
          When the NameNode loads a fsimage,it doesn't care about whether the block is an EC Block during putting the blocks into the BlocksMap.
          But it should take different lookup actions for non-EC/EC blocks during processing BlockReport.
          For non-EC blocks it only does a lookup in the blocksmap by blockid.
          For EC blocks it must convert the blockid to EC-Blockid(the blockid of the first block of a EC block group).

          I think the current implementation has complete this function.

            public BlockInfo getStoredBlock(Block block) {
              BlockInfo info = null;
              if (BlockIdManager.isStripedBlockID(block.getBlockId())) {
                info = blocksMap.getStoredBlock(
                    new Block(BlockIdManager.convertToStripedID(block.getBlockId())));     //AAA
              }
              if (info == null) {
                info = blocksMap.getStoredBlock(block);              // BBB
              }
              return info;
            }
          

          1. A StripedBlockId(whether or not the low 4 bits are not '0000') comes and the block is really EC block, it can be found by AAA.
          2. A StripedBlockId whose low 4 bits are '0000' comes but the block is not EC, it can also be found by AAA.
          3. A StripedBlockId whose low 4 bits are NOT '0000' comes but the block is not EC, it can be found by BBB.
          3. A non-StripedBlockId comes,it can be found by BBB.

          So I don't think we need to process this issue.

          huizane Hui Zheng added a comment - Hi szetszwo I want to confirm my understanding about this issue.Is it correct? When the NameNode loads a fsimage,it doesn't care about whether the block is an EC Block during putting the blocks into the BlocksMap. But it should take different lookup actions for non-EC/EC blocks during processing BlockReport. For non-EC blocks it only does a lookup in the blocksmap by blockid. For EC blocks it must convert the blockid to EC-Blockid(the blockid of the first block of a EC block group). I think the current implementation has complete this function. public BlockInfo getStoredBlock(Block block) { BlockInfo info = null ; if (BlockIdManager.isStripedBlockID(block.getBlockId())) { info = blocksMap.getStoredBlock( new Block(BlockIdManager.convertToStripedID(block.getBlockId()))); //AAA } if (info == null ) { info = blocksMap.getStoredBlock(block); // BBB } return info; } 1. A StripedBlockId(whether or not the low 4 bits are not '0000') comes and the block is really EC block, it can be found by AAA. 2. A StripedBlockId whose low 4 bits are '0000' comes but the block is not EC, it can also be found by AAA. 3. A StripedBlockId whose low 4 bits are NOT '0000' comes but the block is not EC, it can be found by BBB. 3. A non-StripedBlockId comes,it can be found by BBB. So I don't think we need to process this issue.
          szetszwo Tsz-wo Sze added a comment -

          You are right that the current implementation already has this function but it does an additional lookup. Rewrite getStoredBlock(..) as below.

          public BlockInfo getStoredBlock(Block block) {
              if (BlockIdManager.isStripedBlockID(block.getBlockId())) {
                BlockInfo info =  blocksMap.getStoredBlock(
                    new Block(BlockIdManager.convertToStripedID(block.getBlockId())));     //AAA
                if (info == null) {
                  info = blocksMap.getStoredBlock(block);              // BBB1
                }
                return info;
              } else {
                return blocksMap.getStoredBlock(block);              // BBB2
              }
            }
          

          When the input block ID is a striped block ID and not found at AAA, it tries BBB1. We can eliminate BBB1 if we know that there is no non-EC block using a striped block ID.

          This JIRA is to

          1. check whether there is a non-EC block using a striped block ID during fsimage/edit loading.
          2. If yes, use the current logic. Otherwise, skip BBB1. i.e.
            public BlockInfo getStoredBlock(Block block) {
                if (BlockIdManager.isStripedBlockID(block.getBlockId())) {
                  BlockInfo info =  blocksMap.getStoredBlock(
                      new Block(BlockIdManager.convertToStripedID(block.getBlockId())));     //AAA
                  if (info == null && hasNonEcBlockUsingStripID) {
                    info = blocksMap.getStoredBlock(block);              // BBB1
                  }
                  return info;
                } else {
                  return blocksMap.getStoredBlock(block);              // BBB2
                }
              }
            
          szetszwo Tsz-wo Sze added a comment - You are right that the current implementation already has this function but it does an additional lookup. Rewrite getStoredBlock(..) as below. public BlockInfo getStoredBlock(Block block) { if (BlockIdManager.isStripedBlockID(block.getBlockId())) { BlockInfo info = blocksMap.getStoredBlock( new Block(BlockIdManager.convertToStripedID(block.getBlockId()))); //AAA if (info == null ) { info = blocksMap.getStoredBlock(block); // BBB1 } return info; } else { return blocksMap.getStoredBlock(block); // BBB2 } } When the input block ID is a striped block ID and not found at AAA, it tries BBB1. We can eliminate BBB1 if we know that there is no non-EC block using a striped block ID. This JIRA is to check whether there is a non-EC block using a striped block ID during fsimage/edit loading. If yes, use the current logic. Otherwise, skip BBB1. i.e. public BlockInfo getStoredBlock(Block block) { if (BlockIdManager.isStripedBlockID(block.getBlockId())) { BlockInfo info = blocksMap.getStoredBlock( new Block(BlockIdManager.convertToStripedID(block.getBlockId()))); //AAA if (info == null && hasNonEcBlockUsingStripID) { info = blocksMap.getStoredBlock(block); // BBB1 } return info; } else { return blocksMap.getStoredBlock(block); // BBB2 } }
          huizane Hui Zheng added a comment -

          Do you mean we need to optimize the code for skipping the unnecessary lookup if we know there are no non-EC block using a striped block ID?

          huizane Hui Zheng added a comment - Do you mean we need to optimize the code for skipping the unnecessary lookup if we know there are no non-EC block using a striped block ID?
          szetszwo Tsz-wo Sze added a comment -

          Yes, that's correct.

          szetszwo Tsz-wo Sze added a comment - Yes, that's correct.
          huizane Hui Zheng added a comment -

          check whether there is a non-EC block using a striped block ID during fsimage/edit loading.

          I think we also need to keep track of the deletion of the block. It will be a bit complicated and error prone to implement.
          It also cause performance down by counting(in multiple threads) the number of such blocks.
          So it may be better to tolerate the additional lookup.

          huizane Hui Zheng added a comment - check whether there is a non-EC block using a striped block ID during fsimage/edit loading. I think we also need to keep track of the deletion of the block. It will be a bit complicated and error prone to implement. It also cause performance down by counting(in multiple threads) the number of such blocks. So it may be better to tolerate the additional lookup.
          szetszwo Tsz-wo Sze added a comment -

          > ... we also need to keep track of the deletion of the block. ...

          Could you give an example to show the reason for keeping track block deletion?

          szetszwo Tsz-wo Sze added a comment - > ... we also need to keep track of the deletion of the block. ... Could you give an example to show the reason for keeping track block deletion?
          huizane Hui Zheng added a comment -

          Suppose that there are some blocks using StripedBlockID when namenode loads fsimage or editlog,so the "hasNonEcBlockUsingStripID" is "true". But after some time user may delete some or all of these blocks by deleting files. If the number of these blocks is 0, the "hasNonEcBlockUsingStripID" will become "false".

          huizane Hui Zheng added a comment - Suppose that there are some blocks using StripedBlockID when namenode loads fsimage or editlog,so the "hasNonEcBlockUsingStripID" is "true". But after some time user may delete some or all of these blocks by deleting files. If the number of these blocks is 0, the "hasNonEcBlockUsingStripID" will become "false".
          szetszwo Tsz-wo Sze added a comment -

          I see. For simplicity, hasNonEcBlockUsingStripID is only initialized during startup but won't be updated.

          szetszwo Tsz-wo Sze added a comment - I see. For simplicity, hasNonEcBlockUsingStripID is only initialized during startup but won't be updated.
          huizane Hui Zheng added a comment -

          Hi Nicholas
          Thank you for clarification.
          I attached a patch for the purpose.

          huizane Hui Zheng added a comment - Hi Nicholas Thank you for clarification. I attached a patch for the purpose.
          szetszwo Tsz-wo Sze added a comment -

          We should also check the block IDs for adding UnderConstruction files and snapshot deleted files in fsimage and FSEditLogLoader.updateBlocks(..). In general, let's add a utility method to FSImageUtil for addBlockCollection and check block ID there. Then all the calls in fsimage and editlog should use it instead of BlockManager.addBlockCollection.

          szetszwo Tsz-wo Sze added a comment - We should also check the block IDs for adding UnderConstruction files and snapshot deleted files in fsimage and FSEditLogLoader.updateBlocks(..). In general, let's add a utility method to FSImageUtil for addBlockCollection and check block ID there. Then all the calls in fsimage and editlog should use it instead of BlockManager.addBlockCollection.
          huizane Hui Zheng added a comment -

          Thanks szetszwo for reviewing!
          Consider we are using BlockManager to manage hasNonEcBlockUsingStripID,I added a method "addBlockCollectionWithCheck" which do some additional check and update the hasNonEcBlockUsingStripID and reuse "addBlockCollection" into BlockManager.

          huizane Hui Zheng added a comment - Thanks szetszwo for reviewing! Consider we are using BlockManager to manage hasNonEcBlockUsingStripID,I added a method "addBlockCollectionWithCheck" which do some additional check and update the hasNonEcBlockUsingStripID and reuse "addBlockCollection" into BlockManager.
          szetszwo Tsz-wo Sze added a comment -

          +1 patch looks good.

          szetszwo Tsz-wo Sze added a comment - +1 patch looks good.
          szetszwo Tsz-wo Sze added a comment -

          I have committed this. Thanks, Hui!

          szetszwo Tsz-wo Sze added a comment - I have committed this. Thanks, Hui!
          zhz Zhe Zhang added a comment -

          Thanks Hui for the work and Nicholas for the review! This is a nice optimization.

          Sorry for the late comments. Below are some nits that we might want to address in a follow-on task:

          1. Seems to be a little weird to have a setter and a getter with the same name. Since the setter is only called with true, maybe foundxxx instead of hasxxx?
          zhz Zhe Zhang added a comment - Thanks Hui for the work and Nicholas for the review! This is a nice optimization. Sorry for the late comments. Below are some nits that we might want to address in a follow-on task: Seems to be a little weird to have a setter and a getter with the same name. Since the setter is only called with true , maybe foundxxx instead of hasxxx ?
          zhz Zhe Zhang added a comment -

          Sorry clicked Add by mistake. Continuing the comments:

          1. Seems to be a little weird to have a setter and a getter with the same name. Since the setter is only called with true, maybe foundXxx instead of hasXxx?
          2. Since we plan to support EC with non-striping layout in phase II, non-EC blocks should be "contiguous blocks". We can use an abbreviation of contiguous in naming
          3. The following can be a single if with and?
                if (!hasNonEcBlockUsingStripedID()){
                  if (BlockIdManager.isStripedBlockID(block.getBlockId())) {
                    hasNonEcBlockUsingStripedID(true);
                  }
                }
            
          4. Maybe I'm missing something. But what if the block passed to addBlockCollectionWithCheck is a striped block in the fsimage?
          zhz Zhe Zhang added a comment - Sorry clicked Add by mistake. Continuing the comments: Seems to be a little weird to have a setter and a getter with the same name. Since the setter is only called with true, maybe foundXxx instead of hasXxx ? Since we plan to support EC with non-striping layout in phase II, non-EC blocks should be "contiguous blocks". We can use an abbreviation of contiguous in naming The following can be a single if with and ? if (!hasNonEcBlockUsingStripedID()){ if (BlockIdManager.isStripedBlockID(block.getBlockId())) { hasNonEcBlockUsingStripedID( true ); } } Maybe I'm missing something. But what if the block passed to addBlockCollectionWithCheck is a striped block in the fsimage?
          szetszwo Tsz-wo Sze added a comment -

          > ... block passed to addBlockCollectionWithCheck is a striped block ...

          Good point. I will file a JIRA.

          szetszwo Tsz-wo Sze added a comment - > ... block passed to addBlockCollectionWithCheck is a striped block ... Good point. I will file a JIRA.
          szetszwo Tsz-wo Sze added a comment -

          Filed HDFS-8167.

          szetszwo Tsz-wo Sze added a comment - Filed HDFS-8167 .
          huizane Hui Zheng added a comment -

          Thanks zhz and szetszwo
          I will work on HDFS-8167

          huizane Hui Zheng added a comment - Thanks zhz and szetszwo I will work on HDFS-8167
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/)
          HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/ ) HDFS-7994 . Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8548/)
          HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8548/ ) HDFS-7994 . Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/)
          HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/ ) HDFS-7994 . Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/)
          HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/ ) HDFS-7994 . Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/)
          HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/ ) HDFS-7994 . Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/)
          HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/ ) HDFS-7994 . Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/)
          HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/ ) HDFS-7994 . Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java

          People

            huizane Hui Zheng
            szetszwo Tsz-wo Sze
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: