Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6429

MapJoinKey has large memory overhead in typical cases

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.13.0
    • None
    • None

    Description

      The only thing that MJK really needs it hashCode and equals (well, and construction), so there's no need to have array of writables in there. Assuming all the keys for a table have the same structure, for the common case where keys are primitive types, we can store something like a byte array combination of keys to reduce the memory usage. Will probably speed up compares too.

      Attachments

        1. HIVE-6429.WIP.patch
          49 kB
          Sergey Shelukhin
        2. HIVE-6429.patch
          49 kB
          Sergey Shelukhin
        3. HIVE-6429.10.patch
          56 kB
          Sergey Shelukhin
        4. HIVE-6429.09.patch
          59 kB
          Sergey Shelukhin
        5. HIVE-6429.08.patch
          78 kB
          Sergey Shelukhin
        6. HIVE-6429.07.patch
          76 kB
          Sergey Shelukhin
        7. HIVE-6429.06.patch
          76 kB
          Sergey Shelukhin
        8. HIVE-6429.05.patch
          75 kB
          Sergey Shelukhin
        9. HIVE-6429.04.patch
          54 kB
          Sergey Shelukhin
        10. HIVE-6429.03.patch
          51 kB
          Sergey Shelukhin
        11. HIVE-6429.02.patch
          50 kB
          Sergey Shelukhin
        12. HIVE-6429.01.patch
          50 kB
          Sergey Shelukhin

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-6526 clean up BinarySortableSerde a bit

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-6188 Document hive.metastore.try.direct.sql & hive.metastore.try.direct.sql.ddl

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-6430 MapJoin hash table has large memory overhead

          • Major - Major loss of function.
          • Closed

          Activity

            People

              sershe Sergey Shelukhin
              sershe Sergey Shelukhin
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: