[HDFS-8575] Support User level Quota for space and Name (count)

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: nijel
• Created: 11/Jun/15 08:30
• Updated: 13/Oct/15 06:45

Description

I would like to have one feature in HDFS to have quota management at user level.

Background :
When the customer uses a multi tenant solution it will have many Hadoop eco system components like HIVE, HBASE, yarn etc. The base folder of these components are different like /hive - Hive , /hbase -HBase.
Now if a user creates some file or table these will be under the folder specific to component. If the user name is taken into account it looks like

                                /hive/user1/table1
                                /hive/user2/table1
                                /hbase/user1/Htable1
                                /hbase/user2/Htable1
 
Same for yarn/map-reduce data and logs

In this case restricting the user to use a certain amount of disk/file is very difficult since the current quota management is at folder level.

Requirement: User level Quota for space and Name (count). Say user1 can have 100G irrespective of the folder or location used.

Here the idea to consider the file owner ad the key and attribute the quota to it. So the current quota system can have a initial check for the user quota if defined, before validating the folder quota.

Note:
This need a change in fsimage to store the user and quota information

Please have a look on this scenario. If it sounds good, i will create the tasks and the update the design and prototype.

Thanks