[SENTRY-1915] Sentry is doing a lot of work to convert list of paths to HMSPaths structure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Sentry
• Assignee: Alex Kolbasov
• Reporter: Alex Kolbasov
• Created: 02/Sep/17 00:49
• Updated: 20/Nov/17 21:16
• Resolved: 07/Sep/17 00:44

Description

It turns out that in 2.0 we changed the way full snapshots are sent from Sentry to HDFS. Before they were using HMSPaths which used tree structure and eliminated some duplication. Also SENTRY-1827 helped to compressed this on the serialization side.

Now we are using TPathChanges structure that is not tree-based and contains very non-efficient way of representing paths: required list<list<string>> addPaths; so we split each paths on slashes and store list of elements instead of storing a tree. As a result we may use much more memory.

Issue Links

• is related to: SENTRY-1916 Sentry should not store paths outside of the prefix
• is related to: SENTRY-1951 Old SentryStore.retrieveFullPathsImage() should be removed
• relates to: SENTRY-872 Uber jira for HMS HA + Sentry HA redesign
• relates to: SENTRY-1827 Minimize TPathsDump thrift message used in HDFS sync
• relates to: SENTRY-1907 Potential memory optimization when handling big full snapshots.
• relates to: SENTRY-1909 Improvements for memory usage when full path snapshot is sent from Sentry to NN
• links to: Code review
• links to: Review Link for this case