Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20321

Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • None
    • None

    Description

      With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper is bigger than the low Xmx JVMs.

      ***** 64-bit VM: **********************************************************
      org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
       OFFSET  SIZE                                                                     TYPE DESCRIPTION                                  VALUE
            0    16                                                                          (object header)                              N/A
           16     4                                                                      int VectorHashKeyWrapper.hashcode                N/A
           20     4                                                                          (alignment/padding gap)                     
           24     8                                                                   long[] VectorHashKeyWrapper.longValues              N/A
           32     8                                                                 double[] VectorHashKeyWrapper.doubleValues            N/A
           40     8                                                                 byte[][] VectorHashKeyWrapper.byteValues              N/A
           48     8                                                                    int[] VectorHashKeyWrapper.byteStarts              N/A
           56     8                                                                    int[] VectorHashKeyWrapper.byteLengths             N/A
           64     8                   org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] VectorHashKeyWrapper.decimalValues           N/A
           72     8                                                     java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues         N/A
           80     8                 org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] VectorHashKeyWrapper.intervalDayTimeValues   N/A
           88     8                                                                boolean[] VectorHashKeyWrapper.isNull                  N/A
           96     8   org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext VectorHashKeyWrapper.hashCtx                 N/A
      Instance size: 104 bytes
      Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
      

      Pulling this up to a parent class allows for this to be cut down to 32 bytes for the single column case.

      Attachments

        1. HIVE-20321.01.patch
          169 kB
          Matt McCline
        2. HIVE-20321.02.patch
          170 kB
          Matt McCline
        3. HIVE-20321.03.patch
          169 kB
          Matt McCline
        4. HIVE-20321.04.patch
          169 kB
          Matt McCline
        5. HIVE-20321.05.patch
          169 kB
          Matt McCline
        6. HIVE-20321.06.patch
          169 kB
          Matt McCline

        Issue Links

          Activity

            People

              mmccline Matt McCline
              gopalv Gopal Vijayaraghavan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: