[RANGER-2658] Support million policies in ranger.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.0
• Component/s: Ranger
• Assignee: star
• Reporter: star
• Created: 03/Dec/19 06:49
• Updated: 29/Jun/20 10:01

Description

We have nearly a million policies in Tencent. We found many problems by deploying ranger(1.2.0) in such a large scale. It seems that ranger is originally not designed to support large scale policies.

If we would like to support a million policies, the following issues should be emphasized.

   1. Policy import is very slow. It takes us a couple of days to import those policies .

   2. It takes a long time (more than 10s) to search policies by name or user/group in ranger admin. Because it directly go through all policies in cache instead of search from index of database.

   3. It will consume 8GB memory in user process. 

Above problems are relieved in 2.x, but it is not enough.

Issue Links

• relates to: RANGER-2882 Make a abstraction for RangerBasePlugin#isAccessAllowed to allow extension of remote policy evaluation
• relates to: RANGER-2879 Improve performance of incrementally updating policies
• relates to: RANGER-2883 Update RangerServicePoliciesCache incrementally when delete/update/add instead of reloading all policies from the db store
• relates to: RANGER-2884 Search policies from db store instead of filtering by RangerServicePoliciesCache in ranger admin