Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3173 Reduce catalog's memory footprint
  3. IMPALA-3583

Optimize HdfsTable::perPartitionFileDescMap_ to reduce memory usage

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • Impala 2.5.0
    • Impala 2.8.0
    • Catalog

    Description

      HdfsTable::perPartitionFileDescMap_ uses full hdfs path of partition as key

      // Map of file names to file descriptors for each partition location (directory).
      private Map<String, Map<String, FileDescriptor>>
      perPartitionFileDescMap_ = Maps.newHashMap();

      One single entry could take 2.5K bytes to store, among them, there are 5 partition path string.

      Attachments

        Activity

          People

            Unassigned Unassigned
            jyu@cloudera.com Juan Yu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: