[IMPALA-3173] Reduce catalog's memory footprint

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.2.4
• Component/s: Catalog
• Labels: catalog-server performance usability
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Dimitris Tsirogiannis
• Created: 10/Mar/16 23:14
• Updated: 13/Feb/20 18:45

Description

An initial analysis of catalog's heap dumps shows that we can probably reduce it's memory footprint by: a) avoid storing redundant information about catalog entities such as partitions, b) using more compressed data structures.

Currently, for a table with 2 int columns and 1 int partition column and without incremental stats, we use:

• ~930B per partition out of which ~500B are used on hmsParameters_ (<String, String>Map), ~190B on cachedMsPartitionDescriptor_, and ~200B (depending on path) on location.
• ~800B per file descriptor out of which ~530B go to file_blocks and the rest are used for storing the file_name.
• Every HdfsTable also uses two maps that replicate partition locations and file names (e.g. perPartitionFileDescMap_ and nameToPartitionMap_).

A table like that with 100,000 partitions and 10 files per partition requires 1GB and 1.4GB of memory w and w/o incremental stats, respectively.

This is a parent JIRA of IMPALA-2840.

Issue Links

• is a child of: IMPALA-5299 Improve catalog scalability and large catalog handling
• relates to: IMPALA-5990 End-to-end compression of metadata

Sub-Tasks

• Avoid storing redundant information about partitions in the catalog: Open
• Store partition location info with respect to partition keys: Open
• Optimize HdfsTable::perPartitionFileDescMap_ to reduce memory usage: Resolved
• Consider using listLocatedStatus() API to get filestatus and blocklocations in one RPC call: Resolved
• Reduce memory requirements for storing THdfsFileDesc: Resolved
• Prefer binary over string in catalog thrift definitions: Open
• Reduce working memory when processing metadata cache updates: Open