Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14499

Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0, 3.1.4, 3.2.2
    • snapshots
    • None

    Description

      This is the flow of steps where we see a discrepancy between REM_QUOTA and new file operation failure. REM_QUOTA shows a value of  1 but file creation operation does not succeed.

      hdfs@c3265-node3 root$ hdfs dfs -mkdir /dir1
      hdfs@c3265-node3 root$ hdfs dfsadmin -setQuota 2 /dir1
      hdfs@c3265-node3 root$ hdfs dfsadmin -allowSnapshot /dir1
      Allowing snaphot on /dir1 succeeded
      hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
      hdfs@c3265-node3 root$ hdfs dfs -createSnapshot /dir1 snap1
      Created snapshot /dir1/.snapshot/snap1
      hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
      QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME
      2 0 none inf 1 1 0 /dir1
      hdfs@c3265-node3 root$ hdfs dfs -rm /dir1/file1
      19/03/26 11:20:25 INFO fs.TrashPolicyDefault: Moved: 'hdfs://smajetinn/dir1/file1' to trash at: hdfs://smajetinn/user/hdfs/.Trash/Current/dir1/file11553599225772
      hdfs@c3265-node3 root$ hdfs dfs -count -v -q /dir1
      QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME
      2 1 none inf 1 0 0 /dir1
      hdfs@c3265-node3 root$ hdfs dfs -touchz /dir1/file1
      touchz: The NameSpace quota (directories and files) of directory /dir1 is exceeded: quota=2 file count=3

      The issue here, is that the count command takes only files and directories into account not the inode references. When trash is enabled, the deletion of files inside a directory actually does a rename operation as a result of which an inode reference is maintained in the deleted list of the snapshot diff which is taken into account while computing the namespace quota, but count command (getContentSummary()) ,just takes into account just the files and directories, not the referenced entity for calculating the REM_QUOTA. The referenced entity is taken into account for space quota only.

      InodeReference.java:
      -------------------

       @Override
          public final ContentSummaryComputationContext computeContentSummary(
              int snapshotId, ContentSummaryComputationContext summary) {
            final int s = snapshotId < lastSnapshotId ? snapshotId : lastSnapshotId;
            // only count storagespace for WithName
            final QuotaCounts q = computeQuotaUsage(
                summary.getBlockStoragePolicySuite(), getStoragePolicyID(), false, s);
            summary.getCounts().addContent(Content.DISKSPACE, q.getStorageSpace());
            summary.getCounts().addTypeSpaces(q.getTypeSpaces());
            return summary;
          }
      

      Attachments

        1. HDFS-14499.000.patch
          6 kB
          Shashikant Banerjee
        2. HDFS-14499.001.patch
          6 kB
          Shashikant Banerjee
        3. HDFS-14499.002.patch
          6 kB
          Shashikant Banerjee

        Issue Links

          Activity

            People

              shashikant Shashikant Banerjee
              shashikant Shashikant Banerjee
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: