Hive
  1. Hive
  2. HIVE-6518

Add a GC canary to the VectorGroupByOperator to flush whenever a GC is triggered

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: Query Processor
    • Labels:
      None
    • Release Note:
      Flush VectorGroupBy aggregation hashes in case of a full GC

      Description

      The current VectorGroupByOperator implementation flushes the in-memory hashes when the maximum entries or fraction of memory is hit.

      This works for most cases, but there are some corner cases where we hit GC ovehead limits or heap size limits before either of those conditions are reached due to the rest of the pipeline.

      This patch adds a SoftReference as a GC canary. If the soft reference is dead, then a full GC pass happened sometime in the near past & the aggregation hashtables should be flushed immediately before another full GC is triggered.

      1. HIVE-6518.1-tez.patch
        2 kB
        Gopal V
      2. HIVE-6518.2-tez.patch
        3 kB
        Gopal V
      3. HIVE-6518.2.patch
        3 kB
        Gunther Hagleitner
      4. HIVE-6518.3.patch
        3 kB
        Gunther Hagleitner

        Activity

        Jitendra Nath Pandey made changes -
        Fix Version/s 0.13.0 [ 12324986 ]
        Fix Version/s 0.14.0 [ 12326450 ]
        Jitendra Nath Pandey made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.14.0 [ 12326450 ]
        Resolution Fixed [ 1 ]
        Gunther Hagleitner made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Gunther Hagleitner made changes -
        Attachment HIVE-6518.3.patch [ 12634401 ]
        Gunther Hagleitner made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Gopal V made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Gopal V made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Gopal V made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Gopal V made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Gunther Hagleitner made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Gunther Hagleitner made changes -
        Attachment HIVE-6518.2.patch [ 12631991 ]
        Gunther Hagleitner made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Gopal V made changes -
        Attachment HIVE-6518.2-tez.patch [ 12631872 ]
        Gopal V made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note Flush VectorGroupBy aggregation hashes in case of a full GC
        Gopal V made changes -
        Field Original Value New Value
        Attachment HIVE-6518.1-tez.patch [ 12631621 ]
        Gopal V created issue -

          People

          • Assignee:
            Gopal V
            Reporter:
            Gopal V
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development