Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13809

Hybrid Grace Hash Join memory usage estimation didn't take into account the bloom filter size

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 2.1.0
    • Fix Version/s: 2.1.1, 2.2.0
    • Component/s: Hive
    • Labels:
      None

      Description

      Memory estimation is important during hash table loading, because we need to make the decision of whether to load the next hash partition in memory or spill it. If the assumption is there's enough memory but it turns out not the case, we will run into OOM problem.

      Currently hybrid grace hash join memory usage estimation didn't take into account the bloom filter size. In large test cases (TB scale) the bloom filter grows as big as hundreds of MB, big enough to cause estimation error.

      The solution is to count in the bloom filter size into memory estimation.

      Another issue this patch will fix is possible NPE due to object cache reuse during hybrid grace hash join.

        Attachments

        1. HIVE-13809.1.patch
          9 kB
          Wei Zheng

          Issue Links

            Activity

              People

              • Assignee:
                wzheng Wei Zheng
                Reporter:
                wzheng Wei Zheng
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: