Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4642

TestFragmentLifecycle broken; blocking gvos

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      TestFragmentLifecycle has been failing regularly in runs this weekend, blocking gvos and nightly tests.

      At first I thought this was related to IMPALA-4519 , but now I suspect it's not due to a timing issue and instead something has changed causing the test to fail.

      02:17:51 =================================== FAILURES ===================================
      02:17:51 ________________ TestFragmentLifecycle.test_failure_in_prepare _________________
      02:17:51 query_test/test_lifecycle.py:45: in test_failure_in_prepare
      02:17:51     v.wait_for_metric(self.IN_FLIGHT_FRAGMENTS, 0)
      02:17:51 verifiers/metric_verifier.py:59: in wait_for_metric
      02:17:51     self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
      02:17:51 common/impala_service.py:85: in wait_for_metric_value
      02:17:51     (metric_name, expected_value, timeout)
      02:17:51 E   AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 0 in 60s
      02:17:51 ---------------------------- Captured stderr setup -----------------------------
      02:17:51 -- connecting to: localhost:21000
      02:17:51 ----------------------------- Captured stderr call -----------------------------
      02:17:51 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      02:17:51 -- executing against localhost:21000
      02:17:51 SET DEBUG_ACTION='-1:0:PREPARE:FAIL';
      02:17:51 
      02:17:51 -- executing against localhost:21000
      02:17:51 SELECT COUNT(*) FROM functional.alltypes;
      02:17:51 
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51  generated xml file: /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/ee_tests/results/TEST-impala-serial.xml 
      02:17:51 =========================== short test summary info ============================
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mjacobs Matthew Jacobs
            mjacobs Matthew Jacobs
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment