Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3173 Reduce catalog's memory footprint
  3. IMPALA-3583

Optimize HdfsTable::perPartitionFileDescMap_ to reduce memory usage

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Catalog
    • Labels:

      Description

      HdfsTable::perPartitionFileDescMap_ uses full hdfs path of partition as key

      // Map of file names to file descriptors for each partition location (directory).
      private Map<String, Map<String, FileDescriptor>>
      perPartitionFileDescMap_ = Maps.newHashMap();

      One single entry could take 2.5K bytes to store, among them, there are 5 partition path string.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jyu@cloudera.com Juan Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: