Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20649

LLAP aware memory manager for Orc writers

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.2.0, 4.0.0
    • 3.2.0, 4.0.0-alpha-1
    • None
    • None

    Description

      ORC writer has its own memory manager that assumes memory usage or memory available based on JVM heap (MemoryMX bean). This works on tez container mode execution model but not in LLAP where container sizes (and Xmx) are typically high and there are multiple executors per LLAP daemon. This custom memory manager should be aware of memory bounds per executor. 

      Attachments

        1. HIVE-20649.1.patch
          7 kB
          Prasanth Jayachandran
        2. HIVE-20649.2.patch
          8 kB
          Prasanth Jayachandran
        3. HIVE-20649.3.patch
          8 kB
          Prasanth Jayachandran
        4. HIVE-20649.4.patch
          8 kB
          Prasanth Jayachandran
        5. HIVE-20649.5.patch
          8 kB
          Prasanth Jayachandran
        6. HIVE-20649.6.patch
          8 kB
          Prasanth Jayachandran

        Issue Links

          requires

          Bug - A problem which impairs or prevents the functions of the product. ORC-409 Changes for extending MemoryManagerImpl

          • Major - Major loss of function.
          • Closed

          Activity

            People

              prasanth_j Prasanth Jayachandran
              prasanth_j Prasanth Jayachandran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: