[IMPALA-3583] Optimize HdfsTable::perPartitionFileDescMap_ to reduce memory usage

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0
• Fix Version/s: Impala 2.8.0
• Component/s: Catalog
• Labels: catalog-server
• Assignee: Unassigned
• Reporter: Juan Yu
• Created: 20/May/16 19:24
• Updated: 24/Jan/17 04:16
• Resolved: 03/Jan/17 16:52

Description

HdfsTable::perPartitionFileDescMap_ uses full hdfs path of partition as key

// Map of file names to file descriptors for each partition location (directory).
private Map<String, Map<String, FileDescriptor>>
perPartitionFileDescMap_ = Maps.newHashMap();

One single entry could take 2.5K bytes to store, among them, there are 5 partition path string.