Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14504

Rename with Snapshots does not honor quota limit

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.1, 3.4.0
    • None
    • None

    Description

      Steps to Reproduce:

      ----------------------------

      HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2
      
      2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
      2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
      2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      Allowing snapshot on /dir2 succeeded
      
      HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
      2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1
      
      2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      Created snapshot /dir2/.snapshot/snap1
      HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2
      
      2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
      
      2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      Found 1 items
      
      -rw-r--r--   1 sbanerjee hadoop          0 2019-05-21 15:10 /dir2/file2
      
      HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex
      
      2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      touchz: The NameSpace quota (directories and files) of directory /dir2 is exceeded: quota=3 file count=4
      
      HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2
      
      2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      Created snapshot /dir2/.snapshot/snap2
      
      
      HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2
      
      2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      Found 1 items
      
      -rw-r--r--   1 sbanerjee hadoop          0 2019-05-21 15:10 /dir2/file2
      
      HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3
      
      2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey
      
      2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      touchz: The NameSpace quota (directories and files) of directory /dir2 is exceeded: quota=3 file count=5
      

      // create operation fails here as it has already exceeded the quota limit

      HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3
      
      2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      Created snapshot /dir2/.snapshot/snap3
      
      HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4
      
      2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      

      // Rename operation succeeds here adding on to the namespace quota

      HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez
      
      2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      
      touchz: The NameSpace quota (directories and files) of directory /dir2 is exceeded: quota=3 file count=6
      

      // File creation fails here but file count has been increased to 6, bcoz of the previous rename operation

      
      

      The quota being set here is 3. Each successive rename adds an entry to the deleted list of the snapshot diff which gets accounted in the namespace quota, but the rename operation is allowed even when it exceeds the quota limit with snapshots. Once, an attempt is made to create a file, it fails.

      Attachments

        1. HDFS-14504.001.patch
          3 kB
          Hemanth Boyina
        2. HDFS-14504.002.patch
          7 kB
          Hemanth Boyina

        Issue Links

          Activity

            People

              hemanthboyina Hemanth Boyina
              shashikant Shashikant Banerjee
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: