Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7379

test_random_rpc_timeout failed on exhaustive build: Debug webpage did not become available in expected time

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: Impala 3.1.0
    • Fix Version/s: Not Applicable
    • Component/s: Backend
    • Labels:
      None
    • Epic Color:
      ghx-label-1

      Description

      Stacktrace

      custom_cluster/test_rpc_timeout.py:131: in test_random_rpc_timeout
          self.execute_query_verify_metrics(self.TEST_QUERY, 10)
      custom_cluster/test_rpc_timeout.py:51: in execute_query_verify_metrics
          v.wait_for_metric("impala-server.num-fragments-in-flight", 0)
      verifiers/metric_verifier.py:62: in wait_for_metric
          self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
      common/impala_service.py:132: in wait_for_metric_value
          json.dumps(self.read_debug_webpage('memz?json')),
      common/impala_service.py:63: in read_debug_webpage
          return self.open_debug_webpage(page_name, timeout=timeout, interval=interval).read()
      common/impala_service.py:60: in open_debug_webpage
          assert 0, 'Debug webpage did not become available in expected time.'
      E   AssertionError: Debug webpage did not become available in expected time.
      

      Standard Error

      10:27:58 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO
      10:27:58 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
      10:27:59 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO
      10:28:00 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
      10:28:01 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
      10:28:04 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      10:28:04 MainThread: Waiting for num_known_live_backends=3. Current value: 1
      10:28:05 MainThread: Waiting for num_known_live_backends=3. Current value: 2
      10:28:06 MainThread: num_known_live_backends has reached value: 3
      10:28:06 MainThread: num_known_live_backends has reached value: 3
      10:28:06 MainThread: num_known_live_backends has reached value: 3
      10:28:06 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors).
      MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      MainThread: Metric 'statestore.live-backends' has reached desired value: 4
      MainThread: num_known_live_backends has reached value: 3
      MainThread: num_known_live_backends has reached value: 3
      MainThread: num_known_live_backends has reached value: 3
      -- connecting to: localhost:21000
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      -- executing against localhost:21000
      select count(c2.string_col) from      functional.alltypestiny join functional.alltypessmall c2;
      
      MainThread: no process found with pid 19511
      MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage did not become available in expected time.
      MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: None
      MainThread: Sleeping 1s before next retry.
      MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage did not become available in expected time.
      MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: None
      MainThread: Sleeping 1s before next retry.
      MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage did not become available in expected time.
      MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: None
      MainThread: Sleeping 1s before next retry.
      MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage did not become available in expected time.
      MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: None
      MainThread: Sleeping 1s before next retry.
      MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage did not become available in expected time.
      MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: None
      MainThread: Sleeping 1s before next retry.
      MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-ec2-centos74-m5-4xlarge-ondemand-0458.vpc.cloudera.com:25000
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage did not become available in expected time.
      MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: None
      MainThread: Sleeping 1s before next retry.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      MainThread: Debug webpage not yet available.
      

        Attachments

          Activity

            People

            • Assignee:
              kwho Michael Ho
              Reporter:
              dknupp David Knupp
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: