Description
Since random block IDs were supported by some early version of HDFS, the block ID reserved for EC blocks could be already used by some existing blocks in a cluster. During NameNode startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap lookup.
Attachments
Attachments
- HDFS-7994_001.patch
- 9 kB
- Hui Zheng
- HDFS-7994_002.patch
- 19 kB
- Hui Zheng
Activity
Hi szetszwo
I want to confirm my understanding about this issue.Is it correct?
When the NameNode loads a fsimage,it doesn't care about whether the block is an EC Block during putting the blocks into the BlocksMap.
But it should take different lookup actions for non-EC/EC blocks during processing BlockReport.
For non-EC blocks it only does a lookup in the blocksmap by blockid.
For EC blocks it must convert the blockid to EC-Blockid(the blockid of the first block of a EC block group).
I think the current implementation has complete this function.
public BlockInfo getStoredBlock(Block block) { BlockInfo info = null; if (BlockIdManager.isStripedBlockID(block.getBlockId())) { info = blocksMap.getStoredBlock( new Block(BlockIdManager.convertToStripedID(block.getBlockId()))); //AAA } if (info == null) { info = blocksMap.getStoredBlock(block); // BBB } return info; }
1. A StripedBlockId(whether or not the low 4 bits are not '0000') comes and the block is really EC block, it can be found by AAA.
2. A StripedBlockId whose low 4 bits are '0000' comes but the block is not EC, it can also be found by AAA.
3. A StripedBlockId whose low 4 bits are NOT '0000' comes but the block is not EC, it can be found by BBB.
3. A non-StripedBlockId comes,it can be found by BBB.
So I don't think we need to process this issue.
You are right that the current implementation already has this function but it does an additional lookup. Rewrite getStoredBlock(..) as below.
public BlockInfo getStoredBlock(Block block) { if (BlockIdManager.isStripedBlockID(block.getBlockId())) { BlockInfo info = blocksMap.getStoredBlock( new Block(BlockIdManager.convertToStripedID(block.getBlockId()))); //AAA if (info == null) { info = blocksMap.getStoredBlock(block); // BBB1 } return info; } else { return blocksMap.getStoredBlock(block); // BBB2 } }
When the input block ID is a striped block ID and not found at AAA, it tries BBB1. We can eliminate BBB1 if we know that there is no non-EC block using a striped block ID.
This JIRA is to
- check whether there is a non-EC block using a striped block ID during fsimage/edit loading.
- If yes, use the current logic. Otherwise, skip BBB1. i.e.
public BlockInfo getStoredBlock(Block block) { if (BlockIdManager.isStripedBlockID(block.getBlockId())) { BlockInfo info = blocksMap.getStoredBlock( new Block(BlockIdManager.convertToStripedID(block.getBlockId()))); //AAA if (info == null && hasNonEcBlockUsingStripID) { info = blocksMap.getStoredBlock(block); // BBB1 } return info; } else { return blocksMap.getStoredBlock(block); // BBB2 } }
Do you mean we need to optimize the code for skipping the unnecessary lookup if we know there are no non-EC block using a striped block ID?
check whether there is a non-EC block using a striped block ID during fsimage/edit loading.
I think we also need to keep track of the deletion of the block. It will be a bit complicated and error prone to implement.
It also cause performance down by counting(in multiple threads) the number of such blocks.
So it may be better to tolerate the additional lookup.
> ... we also need to keep track of the deletion of the block. ...
Could you give an example to show the reason for keeping track block deletion?
Suppose that there are some blocks using StripedBlockID when namenode loads fsimage or editlog,so the "hasNonEcBlockUsingStripID" is "true". But after some time user may delete some or all of these blocks by deleting files. If the number of these blocks is 0, the "hasNonEcBlockUsingStripID" will become "false".
I see. For simplicity, hasNonEcBlockUsingStripID is only initialized during startup but won't be updated.
Hi Nicholas
Thank you for clarification.
I attached a patch for the purpose.
We should also check the block IDs for adding UnderConstruction files and snapshot deleted files in fsimage and FSEditLogLoader.updateBlocks(..). In general, let's add a utility method to FSImageUtil for addBlockCollection and check block ID there. Then all the calls in fsimage and editlog should use it instead of BlockManager.addBlockCollection.
Thanks szetszwo for reviewing!
Consider we are using BlockManager to manage hasNonEcBlockUsingStripID,I added a method "addBlockCollectionWithCheck" which do some additional check and update the hasNonEcBlockUsingStripID and reuse "addBlockCollection" into BlockManager.
Thanks Hui for the work and Nicholas for the review! This is a nice optimization.
Sorry for the late comments. Below are some nits that we might want to address in a follow-on task:
- Seems to be a little weird to have a setter and a getter with the same name. Since the setter is only called with true, maybe foundxxx instead of hasxxx?
Sorry clicked Add by mistake. Continuing the comments:
- Seems to be a little weird to have a setter and a getter with the same name. Since the setter is only called with true, maybe foundXxx instead of hasXxx?
- Since we plan to support EC with non-striping layout in phase II, non-EC blocks should be "contiguous blocks". We can use an abbreviation of contiguous in naming
- The following can be a single if with and?
if (!hasNonEcBlockUsingStripedID()){ if (BlockIdManager.isStripedBlockID(block.getBlockId())) { hasNonEcBlockUsingStripedID(true); } }
- Maybe I'm missing something. But what if the block passed to addBlockCollectionWithCheck is a striped block in the fsimage?
> ... block passed to addBlockCollectionWithCheck is a striped block ...
Good point. I will file a JIRA.
FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/)
HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8548/)
HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/)
HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/)
HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/)
HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/)
HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/)
HDFS-7994. Detect if resevered EC Block ID is already used during (zhezhang: rev 4c039b0876bb9399c2b4a751ad7b99b36349117b)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
Hi Nicholas
I would like to work on this issue.