Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20760

Reducing memory overhead due to multiple HiveConfs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Configuration
    • None

    Description

      The issue is that every Hive task has to load its own version of HiveConf. When running with a large number of cores per executor (HoS), there is a significant (~10%) amount of memory wasted due to this duplication. 

      I looked into the problem and found a way to reduce the overhead caused by the multiple HiveConf objects.

      I've created an implementation of Properties, somewhat similar to CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve this problem, because it drops the interned Properties right after we add a new property.

      So my implementation looks like this:

      • When we create a new HiveConf from an existing one (copy constructor), we change the properties object stored by HiveConf to the new Properties implementation (HiveConfProperties). We have 2 possible way to do this. Either we change the visibility of the properties field in the ancestor class (Configuration which comes from hadoop) to protected, or a simpler way is to just change the type using reflection.
      • HiveConfProperties instantly intern the given properties. After this, every time we add a new property to HiveConf, we add it to an additional Properties object. This way if we create multiple HiveConf with the same base properties, they will use the same Properties object but each session/task can add its own unique properties.
      • Getting a property from HiveConfProperties would look like this: (I stored the non-interned properties in super class)

                      String property=super.getProperty(key);
                      if (property == null) property= interned.getProperty(key);
                      return property;

      Running some tests showed that the interning works (with 50 connections to HiveServer2, heapdumps created after sessions are created for queries): 

      Overall memory:

               original: 34,599K              interned: 20,582K

      Retained memory of HiveConfs:

              original: 16,366K               interned: 10,804K

      I attach the JXray reports about the heapdumps.

      What are your thoughts about this solution? 

      Attachments

        1. HIVE-20760.10.patch
          29 kB
          Barnabas Maidics
        2. HIVE-20760.11.patch
          29 kB
          Barnabas Maidics
        3. HIVE-20760.12.patch
          31 kB
          Barnabas Maidics
        4. HIVE-20760.13.patch
          31 kB
          Barnabas Maidics
        5. HIVE-20760.4.patch
          23 kB
          Barnabas Maidics
        6. HIVE-20760.5.patch
          28 kB
          Barnabas Maidics
        7. HIVE-20760.6.patch
          28 kB
          Barnabas Maidics
        8. HIVE-20760.7.patch
          27 kB
          Barnabas Maidics
        9. HIVE-20760.8.patch
          27 kB
          Barnabas Maidics
        10. HIVE-20760.9.patch
          29 kB
          Barnabas Maidics
        11. HIVE-20760.patch
          19 kB
          Barnabas Maidics
        12. HIVE-20760-1.patch
          23 kB
          Barnabas Maidics
        13. HIVE-20760-2.patch
          23 kB
          Barnabas Maidics
        14. HIVE-20760-3.patch
          23 kB
          Barnabas Maidics
        15. hiveconf_interned.html
          2.91 MB
          Barnabas Maidics
        16. hiveconf_original.html
          2.56 MB
          Barnabas Maidics

        Activity

          People

            b.maidics Barnabas Maidics
            b.maidics Barnabas Maidics
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: