Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3173 Reduce catalog's memory footprint
  3. IMPALA-3583

Optimize HdfsTable::perPartitionFileDescMap_ to reduce memory usage

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Catalog
    • Labels:

      Description

      HdfsTable::perPartitionFileDescMap_ uses full hdfs path of partition as key

      // Map of file names to file descriptors for each partition location (directory).
      private Map<String, Map<String, FileDescriptor>>
      perPartitionFileDescMap_ = Maps.newHashMap();

      One single entry could take 2.5K bytes to store, among them, there are 5 partition path string.

        Activity

        Hide
        jbapple Jim Apple added a comment -

        This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09.

        2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that:

        https://github.com/apache/incubator-impala/commits/2.8.0

        Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0?

        Thank you!

        Show
        jbapple Jim Apple added a comment - This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09. 2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that: https://github.com/apache/incubator-impala/commits/2.8.0 Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0? Thank you!
        Hide
        bharathv bharath v added a comment -

        IMPALA-4172 removed this map.

        Show
        bharathv bharath v added a comment - IMPALA-4172 removed this map.
        Hide
        mmokhtar Mostafa Mokhtar added a comment -

        Juan Yu
        Do you have a breakdown of memory consumers or a heap dump?

        Show
        mmokhtar Mostafa Mokhtar added a comment - Juan Yu Do you have a breakdown of memory consumers or a heap dump?

          People

          • Assignee:
            Unassigned
            Reporter:
            jyu@cloudera.com Juan Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development