[HDFS-14504] Rename with Snapshots does not honor quota limit

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.4.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: snapshots
• Target Version/s: 3.3.1, 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Hemanth Boyina
• Reporter: Shashikant Banerjee
• Created: 21/May/19 10:31
• Updated: 11/Feb/24 12:59
• Resolved: 23/Aug/20 10:45

Description

Steps to Reproduce:

----------------------------

HW15685:bin sbanerjee$ ./hdfs dfs -mkdir /dir2

2019-05-21 15:08:41,615 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

HW15685:bin sbanerjee$ ./hdfs dfsadmin -setQuota 3 /dir2
2019-05-21 15:08:57,326 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

HW15685:bin sbanerjee$ ./hdfs dfsadmin -allowSnapshot /dir2
2019-05-21 15:09:47,239 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Allowing snapshot on /dir2 succeeded

HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/file1
2019-05-21 15:10:01,573 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap1

2019-05-21 15:10:16,332 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Created snapshot /dir2/.snapshot/snap1
HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file1 /dir2/file2

2019-05-21 15:10:49,292 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2

2019-05-21 15:11:05,207 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Found 1 items

-rw-r--r--   1 sbanerjee hadoop          0 2019-05-21 15:10 /dir2/file2

HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filex

2019-05-21 15:11:43,765 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

touchz: The NameSpace quota (directories and files) of directory /dir2 is exceeded: quota=3 file count=4

HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap2

2019-05-21 15:12:05,464 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Created snapshot /dir2/.snapshot/snap2


HW15685:bin sbanerjee$ ./hdfs dfs -ls /dir2

2019-05-21 15:12:25,072 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Found 1 items

-rw-r--r--   1 sbanerjee hadoop          0 2019-05-21 15:10 /dir2/file2

HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file2 /dir2/file3

2019-05-21 15:12:35,908 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filey

2019-05-21 15:12:49,998 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

touchz: The NameSpace quota (directories and files) of directory /dir2 is exceeded: quota=3 file count=5

// create operation fails here as it has already exceeded the quota limit

HW15685:bin sbanerjee$ ./hdfs dfs -createSnapshot /dir2 snap3

2019-05-21 15:13:07,656 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Created snapshot /dir2/.snapshot/snap3

HW15685:bin sbanerjee$ ./hdfs dfs -mv /dir2/file3 /dir2/file4

2019-05-21 15:13:20,715 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

// Rename operation succeeds here adding on to the namespace quota

HW15685:bin sbanerjee$ ./hdfs dfs -touchz /dir2/filez

2019-05-21 15:13:30,486 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

touchz: The NameSpace quota (directories and files) of directory /dir2 is exceeded: quota=3 file count=6

// File creation fails here but file count has been increased to 6, bcoz of the previous rename operation

The quota being set here is 3. Each successive rename adds an entry to the deleted list of the snapshot diff which gets accounted in the namespace quota, but the rename operation is allowed even when it exceeds the quota limit with snapshots. Once, an attempt is made to create a file, it fails.

Issue Links

• contains: HDFS-14505 "touchz" command should check quota limit before deleting an already existing file