[SENTRY-1687] FullUpdateInitializer can be more efficient

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Sentry
• Assignee: Alex Kolbasov
• Reporter: Alex Kolbasov
• Created: 01/Apr/17 00:41
• Updated: 24/Jul/17 14:51
• Resolved: 26/Apr/17 18:15

Description

The FullUpdateInitializer follows the MetastoreCacheInitializer. It reads a bunch of information from HMS and uses Thrift structures to pass around, but in the end it just constructs Map<String, Set<String>>. It uses concurrent fetches from HMS, but synchronizes a lot on common data structures to update them.

I think that we can refactor all this code to make it faster and consume less memory. The idea is the following:

Use background threads to collect Thrift results from HMS calls (database, table and partition data). Then we can use a single thread to construct the resulting update and return it without using intermediate Thrift methods.

Issue Links

• is related to: SENTRY-1606 Sentry HDFS Sync should survive in presence of bad paths objects
• is related to: SENTRY-1698 PathsUpdate.parsePath() calls FileSystem.getDefaultUri() way too often
• is related to: SENTRY-1700 FullUpdateInitializer should not use preconditions to verify HMS data