Details
Description
The NameNode manages the global state of the cluster. We should always take NameNode's records as the sole criterion because no matter what inconsistent is the NameNode should finally make everything right based on it's records. Let's call it rule NSC(NameNode is the Sole Criterion). That means when we do all quota related rpcs, we do the quota check according to NameNode's records regardless of any inconsistent situation, such as the replicas doesn't match the storage policy of the file, or the replicas count doesn't match the file's set replica.
The work SPS deals with the wrongly palced replicas. There is a thought about putting off the consume update of the DirectoryQuota until all replicas are re-placed by SPS. I can't agree that because if we do so we will abandon letting the NameNode's records to be the sole criterion. The block replication is a good example of the rule NSC. When we count the consume of a file(CONTIGUOUS), we multiply the replication factor with the file's length, no matter the blocks are under replicated or excessed. We should do the same thing for the storage type quota.
Another concern is the change will let setStoragePolicy throw QuotaByStorageTypeExceededException which it doesn't before. I don't think it's a big problem since the setStoragePolicy already throws IOException. Or we can wrap the QuotaByStorageTypeExceededException in an IOException, but I won't recommend that because it's ugly.
To make storage type consume follow the rule NSC, we need change rename(moving a file with storage policy inherited from it's parent) and setStoragePolicy.
Attachments
Attachments
Issue Links
- is a child of
-
HDFS-14566 Quota improvement in HDFS
-
- Open
-
- is a parent of
-
-
- Patch Available
-