Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27695

Intermittent OOM when running TestMiniTezCliDriver

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.0.0-beta-1
    • 4.0.0
    • Test

    Description

      Running all the tests under TestMiniTezCliDriver very frequently (but still intermittently) leads to OutOfMemory errors.

      cd itests/qtest && mvn test -Dtest=TestMiniTezCliDriver

      I set -XX:+HeapDumpOnOutOfMemoryError and the respective heapdumps are attached to this ticket.

      The OOM is thrown from the application master and a quick inspection of the dumps shows that it comes mainly from the accumulation of Configuration objects (~1MB each) by various classes.

      The max heap size for application master is pretty low (~100MB) so it is quite easy to reach. The heap size is explicitly very low for testing purposes but maybe we should re-evaluate the current configurations for the tests.

      Attachments

        1. dag_am_debug_bundles.tar.xz
          17.36 MB
          Stamatis Zampetakis
        2. leak_suspect_1.png
          186 kB
          Stamatis Zampetakis
        3. am_heap_dumps.tar.xz
          36.43 MB
          Stamatis Zampetakis

        Issue Links

          fixes

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-27182 tez_union_with_udf.q with TestMiniTezCliDriver is flaky

          • Major - Major loss of function.
          • Closed
          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-27916 Increase tez.am.resource.memory.mb for TestIcebergCliDrver to 512MB

          • Major - Major loss of function.
          • Closed
          is related to

          Sub-task - The sub-task of the issue HIVE-27718 TestMiniTezCliDriver: save application logs for failed tests

          • Major - Major loss of function.
          • In Progress

          Bug - A problem which impairs or prevents the functions of the product. HIVE-26828 Fix OOM for hybridgrace_hashjoin_2.q

          • Major - Major loss of function.
          • Closed

          Test - A new unit, integration or system test. HIVE-26820 Disable hybridgrace_hashjoin_2.q flaky test

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HIVE-27719 Save heapdump in case of OOM

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #4792

          Activity

            People

              zabetak Stamatis Zampetakis
              zabetak Stamatis Zampetakis
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: