[SENTRY-178] Poor performance for Sentry Policy Service as #of privileges is scaled up

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Assignee: Arun Suresh
• Reporter: Lenni Kuff
• Created: 22/Apr/14 20:03
• Updated: 05/Jun/14 22:02
• Resolved: 21/May/14 20:26

Description

I have observed that as the number of role privileges is scaled up, the performance of the Sentry Service (time it takes to execute a grant/revoke RPC) gets increasingly worse.

The following is how long it takes to execute an RPC to grant/revoke a privilege from a role:

# of Role Privileges (each on different tables)

100 privileges ~2 RPCs/sec
1000 privileges ~1.5 RPCs/sec
2000 privileges - ~.5 RPCs/sec
4000 privileges - ~.2 RPCs/sec

Configuration:
- Sentry Policy Service -> Postgres Backend DB

This means the time to actually execute one grant/revoke RPC using a policy that is securing 4000 tables is >5s.

I tried scaling up the number of clients, but that doesn't appear to improve the throughput since there is a lot of locking that is happening inside the Sentry Policy Service.