[HIVE-9736] StorageBasedAuthProvider should batch namenode-calls where possible.

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.1
• Component/s: Metastore, Security
• Labels: TODOC1.2
• Assignee: Mithun Radhakrishnan
• Reporter: Mithun Radhakrishnan
• Created: 20/Feb/15 02:56
• Updated: 13/Dec/19 15:12

Description

Consider a table partitioned by 2 keys (dt, region). Say a dt partition could have 10000 associated regions. Consider that the user does:

ALTER TABLE my_table DROP PARTITION (dt='20150101');

As things stand now, StorageBasedAuthProvider will make individual DistributedFileSystem.listStatus() calls for each partition-directory, and authorize each one separately. It'd be faster to batch the calls, and examine multiple FileStatus objects at once.

Issue Links

• depends upon: HIVE-9681 Extend HiveAuthorizationProvider to support partition-sets.
• is depended upon by: HIVE-9629 HCatClient.dropPartitions() needs speeding up.
• relates to: HIVE-10223 Consolidate several redundant FileSystem API calls.
• links to: ReviewBoard: r/31615