Uploaded image for project: 'Sentry'
  1. Sentry
  2. SENTRY-1811

Optimize data structures used in HDFS sync

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.8.0, 2.0.0
    • Fix Version/s: 1.8.0, 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      We obtained a heap dump taken from the JVM running Hive Metastore at the time when Sentry HDFS sync operation was performed. I've analyzed this dump with jxray (www.jxray.com) and found that more than 19% of memory is wasted due to empty or suboptimally-sized Java collections:

      9. BAD COLLECTIONS
      
      Total collections: 54,057,249  Bad collections: 31,569,606  Overhead: 5,292,821K (19.3%)
      

      Most of these collections come from thrift classes used by the Sentry plugin, see below. The associated memory waste can be significantly reduced or eliminated if these collections were allocated lazily and then with the initial capacity smaller than the default 16 elements for HashMap/HashSet.

        1,869,023K (6.8%): j.u.HashSet: 3388670 of 1-elem 979,537K (3.6%), 5897806 of empty 552,919K (2.0%), 1010321 of small 336,566K (1.2%)
           <-- org.apache.sentry.hdfs.service.thrift.TPathEntry.children <--  {j.u.HashMap}.values <-- org.apache.sentry.hdfs.service.thrift.TPathsDump.nodeMap <-- org.apache.sentry.hdfs.service.thrift.TPathsUpdate.pathsDump <-- Java Local@7fea0851c360 (org.apache.sentry.hdfs.service.thrift.TPathsUpdate)
        1,190,050K (4.3%): j.u.HashMap: 3382765 of 1-elem 898,546K (3.3%), 1005341 of small 291,503K (1.1%)
           <-- org.apache.sentry.hdfs.HMSPaths$Entry.children <-- org.apache.sentry.hdfs.HMSPaths$Entry.{parent} <--  {j.u.HashSet} <--  {j.u.TreeMap}.values <-- org.apache.sentry.hdfs.HMSPaths.authzObjToPath <-- org.apache.sentry.hdfs.UpdateableAuthzPaths.paths <-- org.apache.sentry.hdfs.MetastorePlugin.authzPaths <-- Java Local@7fe4fe84e030 (org.apache.sentry.hdfs.MetastorePlugin)
        969,442K (3.5%): j.u.TreeSet: 5907188 of 1-elem 969,148K (3.5%)
           <-- org.apache.sentry.hdfs.service.thrift.TPathEntry.authzObjs <--  {j.u.HashMap}.values <-- org.apache.sentry.hdfs.service.thrift.TPathsDump.nodeMap <-- org.apache.sentry.hdfs.service.thrift.TPathsUpdate.pathsDump <-- Java Local@7fea0851c360 (org.apache.sentry.hdfs.service.thrift.TPathsUpdate)
        487,690K (1.8%): j.u.TreeSet: 4801877 of empty 487,690K (1.8%)
           <-- org.apache.sentry.hdfs.HMSPaths$Entry.authzObjs <-- org.apache.sentry.hdfs.HMSPaths$Entry.{parent} <--  {j.u.HashSet} <--  {j.u.TreeMap}.values <-- org.apache.sentry.hdfs.HMSPaths.authzObjToPath <-- org.apache.sentry.hdfs.UpdateableAuthzPaths.paths <-- org.apache.sentry.hdfs.MetastorePlugin.authzPaths <-- Java Local@7fe4fe84e030 (org.apache.sentry.hdfs.MetastorePlugin)
        415,064K (1.5%): j.u.HashMap: 5897806 of empty 414,689K (1.5%)
           <-- org.apache.sentry.hdfs.HMSPaths$Entry.children <--  {j.u.HashSet} <--  {j.u.TreeMap}.values <-- org.apache.sentry.hdfs.HMSPaths.authzObjToPath <-- org.apache.sentry.hdfs.UpdateableAuthzPaths.paths <-- org.apache.sentry.hdfs.MetastorePlugin.authzPaths <-- Java Local@7fe4fe84e030 (org.apache.sentry.hdfs.MetastorePlugin)
      

      Additionally, a significant percentage of memory is wasted due to duplicate strings:

      7. DUPLICATE STRINGS
      
      Total strings: 29,986,017  Unique strings: 9,640,413  Duplicate values: 4,897,743  Overhead: 2,570,746K (9.4%)
      

      Of them, more than 1/3 come from sentry:

        917,331K (3.3%), 10517636 dup strings (498477 unique), 10517636 dup backing arrays:
           <-- org.apache.sentry.hdfs.service.thrift.TPathEntry.pathElement <--  {j.u.HashMap}.values <-- org.apache.sen
      try.hdfs.service.thrift.TPathsDump.nodeMap <-- org.apache.sentry.hdfs.service.thrift.TPathsUpdate.pathsDump <-- Ja
      va Local@7fea0851c360 (org.apache.sentry.hdfs.service.thrift.TPathsUpdate)
      

      These can be eliminated by inserting String.intern() calls in the appropriate places.

        Attachments

        1. SENTRY-1811.03.patch
          16 kB
          Misha Dmitriev
        2. SENTRY-1811.02-sentry-ha-redesign.patch
          15 kB
          Alexander Kolbasov
        3. SENTRY-1811.02.patch
          15 kB
          Misha Dmitriev
        4. SENTRY-1811.01.patch
          15 kB
          Misha Dmitriev

          Issue Links

            Activity

              People

              • Assignee:
                misha@cloudera.com Misha Dmitriev
                Reporter:
                misha@cloudera.com Misha Dmitriev
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: