[SENTRY-2302] Optimize HDFS ACL Synchronization.

• Type: Task
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1
• Component/s: Sentry
• Assignee: Krishna Kalyan
• Reporter: Krishna Kalyan
• Created: 10/Jul/18 13:17
• Updated: 07/Jan/19 19:18

Description

Optimize the time taken for HDFS-SYNC full snapshot.

1. Decrease total time taken for HDFS ACL synchronization.
2. Reduce/Avoid the duration HMS threads are blocked by synchronizing the notifications with sentry.

Here are the time intensive operations in the hdfs-acl synchronization ordered by time taken.

1. Fetching full snapshot from HMS.
2. Persisting the snapshot in the database.
3. Retrieving full snapshot from database.

Sub-Tasks

• Optimize the performence by tuning sentry and HMS properties: Open
• Optimize the time taken for HMS snapshot creation.: Open
• Optimize time taken for persistence HMS snapshot by persisting in parallel: Patch Available
• Reduce the HMS snapshot size by ignoring partitions at default locations: Patch Available
• Avoid HMS event synchronization while sentry is fetching full snapshot.: Resolved
• Filter the path information in the fullupdate send to sentry namenode plug-in: Open
• Create a cache of Path and Perm Full Updates to send to NN: Patch Available
• Minimize the time sentry blocks HMS threads: Patch Available
• Increase the allocation size for auto-increment of id's for Snapshot tables.: Resolved
• Avoid getting the complete partition objects and fetch the location information only.: Open
• Persist HMS paths in batches.: Resolved