Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4642

TestFragmentLifecycle broken; blocking gvos

    Details

      Description

      TestFragmentLifecycle has been failing regularly in runs this weekend, blocking gvos and nightly tests.

      At first I thought this was related to IMPALA-4519 , but now I suspect it's not due to a timing issue and instead something has changed causing the test to fail.

      02:17:51 =================================== FAILURES ===================================
      02:17:51 ________________ TestFragmentLifecycle.test_failure_in_prepare _________________
      02:17:51 query_test/test_lifecycle.py:45: in test_failure_in_prepare
      02:17:51     v.wait_for_metric(self.IN_FLIGHT_FRAGMENTS, 0)
      02:17:51 verifiers/metric_verifier.py:59: in wait_for_metric
      02:17:51     self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
      02:17:51 common/impala_service.py:85: in wait_for_metric_value
      02:17:51     (metric_name, expected_value, timeout)
      02:17:51 E   AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 0 in 60s
      02:17:51 ---------------------------- Captured stderr setup -----------------------------
      02:17:51 -- connecting to: localhost:21000
      02:17:51 ----------------------------- Captured stderr call -----------------------------
      02:17:51 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      02:17:51 -- executing against localhost:21000
      02:17:51 SET DEBUG_ACTION='-1:0:PREPARE:FAIL';
      02:17:51 
      02:17:51 -- executing against localhost:21000
      02:17:51 SELECT COUNT(*) FROM functional.alltypes;
      02:17:51 
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51  generated xml file: /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/ee_tests/results/TEST-impala-serial.xml 
      02:17:51 =========================== short test summary info ============================
      

        Activity

        Hide
        jbapple Jim Apple added a comment -

        This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09.

        2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that:

        https://github.com/apache/incubator-impala/commits/2.8.0

        Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0?

        Thank you!

        Show
        jbapple Jim Apple added a comment - This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09. 2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that: https://github.com/apache/incubator-impala/commits/2.8.0 Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0? Thank you!
        Hide
        mjacobs Matthew Jacobs added a comment -

        commit 73e41cea196703701d40cc67f919287fb3511b9b
        Author: Matthew Jacobs <mj@cloudera.com>
        Date: Mon Dec 12 14:32:07 2016 -0800

        IMPALA-4642: Fix TestFragmentLifecycle failures; kudu test must wait

        Fixes test failures in TestFragmentLifecycle when it runs
        after TestKuduMemLimits which takes some time for all
        fragments to finish closing, even though the query is
        finished. TestFragmentLifecycle checks that there are no
        fragments in flight. For now, this fixes the tests by
        forcing TestKuduMemLimits to wait for all 'in flight'
        fragments to complete before continuing. We still need to
        understand why the KuduScanNode/KuduScanner is taking so
        long to Close() (see IMPALA-4654).

        Change-Id: Ia655a37ff06e92cc55ba05f01d5e94fe39447c65
        Reviewed-on: http://gerrit.cloudera.org:8080/5481
        Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
        Reviewed-by: Henry Robinson <henry@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        mjacobs Matthew Jacobs added a comment - commit 73e41cea196703701d40cc67f919287fb3511b9b Author: Matthew Jacobs <mj@cloudera.com> Date: Mon Dec 12 14:32:07 2016 -0800 IMPALA-4642 : Fix TestFragmentLifecycle failures; kudu test must wait Fixes test failures in TestFragmentLifecycle when it runs after TestKuduMemLimits which takes some time for all fragments to finish closing, even though the query is finished. TestFragmentLifecycle checks that there are no fragments in flight. For now, this fixes the tests by forcing TestKuduMemLimits to wait for all 'in flight' fragments to complete before continuing. We still need to understand why the KuduScanNode/KuduScanner is taking so long to Close() (see IMPALA-4654 ). Change-Id: Ia655a37ff06e92cc55ba05f01d5e94fe39447c65 Reviewed-on: http://gerrit.cloudera.org:8080/5481 Reviewed-by: Sailesh Mukil <sailesh@cloudera.com> Reviewed-by: Henry Robinson <henry@cloudera.com> Tested-by: Internal Jenkins
        Hide
        mjacobs Matthew Jacobs added a comment -

        Filed IMPALA-4645 to track the underlying issue which will still need to be solved.

        Show
        mjacobs Matthew Jacobs added a comment - Filed IMPALA-4645 to track the underlying issue which will still need to be solved.
        Hide
        mjacobs Matthew Jacobs added a comment -

        Reverting the Kudu version bump seems to get the tests to pass, though I think there's an underlying query lifecycle bug that a change in Kudu exposed.

        I observed that while running TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan, the select query used in the test appears to return successfully even though a fragment continues to run. Likely there is a bug in the KuduScanNode/KuduScanner and/or fragment management. I'll open a separate JIRA for that and unblock these tests by reverting the Kudu version bump (commit 39017adffc0cb13ebbdab89902d8403faf9a2b4b).

        Show
        mjacobs Matthew Jacobs added a comment - Reverting the Kudu version bump seems to get the tests to pass, though I think there's an underlying query lifecycle bug that a change in Kudu exposed. I observed that while running TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan, the select query used in the test appears to return successfully even though a fragment continues to run. Likely there is a bug in the KuduScanNode/KuduScanner and/or fragment management. I'll open a separate JIRA for that and unblock these tests by reverting the Kudu version bump (commit 39017adffc0cb13ebbdab89902d8403faf9a2b4b).
        Hide
        mjacobs Matthew Jacobs added a comment -

        My new theory is that this is in fact related to a Kudu change (despite this happening at the same time as the query-wide execution state which initially looked suspicious). If my private run w/ the older kudu version succeeds I will revert the Impala build to the previous Kudu version.

        Show
        mjacobs Matthew Jacobs added a comment - My new theory is that this is in fact related to a Kudu change (despite this happening at the same time as the query-wide execution state which initially looked suspicious). If my private run w/ the older kudu version succeeds I will revert the Impala build to the previous Kudu version.
        Hide
        mjacobs Matthew Jacobs added a comment -

        This seems to be happening constantly on build machines. I'm testing reverting my kudu version bump to try to eliminate that variable.

        Show
        mjacobs Matthew Jacobs added a comment - This seems to be happening constantly on build machines. I'm testing reverting my kudu version bump to try to eliminate that variable.
        Hide
        mjacobs Matthew Jacobs added a comment -

        Marcel Kornacker I suspect this is related to "IMPALA-4014: Introduce query-wide execution state." but I cannot repro this in my environment.

        Show
        mjacobs Matthew Jacobs added a comment - Marcel Kornacker I suspect this is related to " IMPALA-4014 : Introduce query-wide execution state." but I cannot repro this in my environment.

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            mjacobs Matthew Jacobs
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development