Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4659

TestScannersFuzzing should set a mem_limit

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Infrastructure
    • Labels:

      Description

      I saw a few tests fail with Memory Limit Exceeded around the same time on an exhaustive test run: TestHdfsQueries.test_analytic_fns, TestScratchLimit.test_without_specifying_scratch_limit, TestScannersAllTableFormatsWithLimit.test_limit

      They were run concurrently with TestScannersFuzzing. My hypothesis is that some of the instances of the TestScannersFuzzing tests used too much memory and caused the other tests to fail. We should set a query mem_limit on those tests to avoid this happening.

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-4659: fuzz test fixes

        • Apply a 512m mem_limit to all fuzz tests. This limits aggregate memory
          consumption to ~5GB per daemon(assuming 10 concurrent tests).
        • Refactor the exec option handling to use the exec_option dimension.
          This avoids executing the test multiple times redundantly
        • Remove the xfails to reduce noise, since there is no immediate plan to
          fix the product bugs. Instead just pass the tests.

        Testing:
        Ran in a loop for ~1h to flush out flakiness.

        Change-Id: Ie1942ceef252ec3e6171a0a54722b66a7d9abbd7
        Reviewed-on: http://gerrit.cloudera.org:8080/5502
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-4659 : fuzz test fixes Apply a 512m mem_limit to all fuzz tests. This limits aggregate memory consumption to ~5GB per daemon(assuming 10 concurrent tests). Refactor the exec option handling to use the exec_option dimension. This avoids executing the test multiple times redundantly Remove the xfails to reduce noise, since there is no immediate plan to fix the product bugs. Instead just pass the tests. Testing: Ran in a loop for ~1h to flush out flakiness. Change-Id: Ie1942ceef252ec3e6171a0a54722b66a7d9abbd7 Reviewed-on: http://gerrit.cloudera.org:8080/5502 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins
        Hide
        jbapple Jim Apple added a comment -

        This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09.

        2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that:

        https://github.com/apache/incubator-impala/commits/2.8.0

        Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0?

        Thank you!

        Show
        jbapple Jim Apple added a comment - This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09. 2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that: https://github.com/apache/incubator-impala/commits/2.8.0 Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0? Thank you!

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development