Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14633

The StorageType quota and consume in QuotaFeature is not handled for rename

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The NameNode manages the global state of the cluster. We should always take NameNode's records as the sole criterion because no matter what inconsistent is the NameNode should finally make everything right based on it's records. Let's call it rule NSC(NameNode is the Sole Criterion). That means when we do all quota related rpcs, we do the quota check according to NameNode's records regardless of any inconsistent situation, such as the replicas doesn't match the storage policy of the file, or the replicas count doesn't match the file's set replica.
      The work SPS deals with the wrongly palced replicas. There is a thought about putting off the consume update of the DirectoryQuota until all replicas are re-placed by SPS. I can't agree that because if we do so we will abandon letting the NameNode's records to be the sole criterion. The block replication is a good example of the rule NSC. When we count the consume of a file(CONTIGUOUS), we multiply the replication factor with the file's length, no matter the blocks are under replicated or excessed. We should do the same thing for the storage type quota.
      Another concern is the change will let setStoragePolicy throw QuotaByStorageTypeExceededException which it doesn't before. I don't think it's a big problem since the setStoragePolicy already throws IOException. Or we can wrap the QuotaByStorageTypeExceededException in an IOException, but I won't recommend that because it's ugly.
      To make storage type consume follow the rule NSC, we need change rename(moving a file with storage policy inherited from it's parent) and setStoragePolicy. 

        Attachments

        1. HDFS-14633.002.patch
          8 kB
          Xiaoyu Yao
        2. HDFS-14633.003.patch
          5 kB
          Xiaoyu Yao
        3. HDFS-14633.004.patch
          15 kB
          Jinglun
        4. HDFS-14633.005.patch
          5 kB
          Jinglun
        5. HDFS-14633.006.patch
          5 kB
          Jinglun
        6. HDFS-14633.007.patch
          5 kB
          Jinglun
        7. HDFS-14633-testcases-explanation
          4 kB
          Jinglun

          Issue Links

            Activity

              People

              • Assignee:
                LiJinglun Jinglun
                Reporter:
                LiJinglun Jinglun
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: