Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4818

TestCancellationSerial.test_cancel_insert is meta-flaky

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Infrastructure
    • Labels:

      Description

      Found by David Knupp: TestCancellationSerial.test_cancel_insert is so flaky that it doesn't even run the same number of test cases each time:

      jbapple@jbapple-optiplex:~/Impala/tests:((detached from gerrit-asf/master))$ impala-py.test --exploration_strategy=core --collect-only query_test/test_cancellation.py | grep -c insert
      7
      jbapple@jbapple-optiplex:~/Impala/tests:((detached from gerrit-asf/master))$ impala-py.test --exploration_strategy=core --collect-only query_test/test_cancellation.py | grep -c insert
      4
      jbapple@jbapple-optiplex:~/Impala/tests:((detached from gerrit-asf/master))$ impala-py.test --exploration_strategy=core --collect-only query_test/test_cancellation.py | grep -c insert
      3
      jbapple@jbapple-optiplex:~/Impala/tests:((detached from gerrit-asf/master))$ impala-py.test --exploration_strategy=core --collect-only query_test/test_cancellation.py | grep -c insert
      8
      jbapple@jbapple-optiplex:~/Impala/tests:((detached from gerrit-asf/master))$ impala-py.test --exploration_strategy=core --collect-only query_test/test_cancellation.py | grep -c insert
      7
      

      http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/657 and http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/661 ran the same code:

      GIT_HASH: 3426a04952eb5ee906b732b4644ce77c0d24fa7e

      But they have different numbers of test_cancel_insert runs: #657 has 9 and #661 has 5, leading to #657 reporting:

      =============== 1624 tests deselected by "-m 'execute_serially'" ===============
       187 passed, 8 skipped, 1624 deselected, 1 xfailed, 24 warnings in 1944.18 seconds 
      

      While #661 reports:

      =============== 1624 tests deselected by "-m 'execute_serially'" ===============
       183 passed, 8 skipped, 1624 deselected, 1 xfailed, 24 warnings in 1958.31 seconds 
      

        Activity

        Hide
        jbapple Jim Apple added a comment -

        This might be it:

        from random import choice
        
              cls.TestMatrix.add_constraint(lambda v: v.get_value('query') ==\
                  choice(QUERIES.keys()))
        
        Show
        jbapple Jim Apple added a comment - This might be it: from random import choice cls.TestMatrix.add_constraint(lambda v: v.get_value('query') ==\ choice(QUERIES.keys()))
        Hide
        jbapple Jim Apple added a comment -
        Show
        jbapple Jim Apple added a comment - Patch for review: https://gerrit.cloudera.org/#/c/5784/
        Hide
        alex.behm Alexander Behm added a comment -

        Jim Apple, this is fixed right?

        Show
        alex.behm Alexander Behm added a comment - Jim Apple , this is fixed right?
        Hide
        jbapple Jim Apple added a comment -

        IMPALA-4818: Ensure the same number of tests are run every time

        This is a partial revert of Change-Id: I04b4d6db508a26a1a2e4b972b...
        from January 2014. That commit caused test_cancel_insert to run
        different random tests each time it was run; the number of tests run
        could vary between 1 and 9. This commit fixes the number at 9, and the
        same 9 every time, to avoid test flakiness.

        Change-Id: I22cecfbe7c9a102f788d01eb80aa188579ef6d7e
        Reviewed-on: http://gerrit.cloudera.org:8080/5784
        Reviewed-by: Michael Brown <mikeb@cloudera.com>
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        jbapple Jim Apple added a comment - IMPALA-4818 : Ensure the same number of tests are run every time This is a partial revert of Change-Id: I04b4d6db508a26a1a2e4b972b... from January 2014. That commit caused test_cancel_insert to run different random tests each time it was run; the number of tests run could vary between 1 and 9. This commit fixes the number at 9, and the same 9 every time, to avoid test flakiness. Change-Id: I22cecfbe7c9a102f788d01eb80aa188579ef6d7e Reviewed-on: http://gerrit.cloudera.org:8080/5784 Reviewed-by: Michael Brown <mikeb@cloudera.com> Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            jbapple Jim Apple
            Reporter:
            jbapple Jim Apple
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development