[IMPALA-9209] TestRPCException.test_end_data_stream_error is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.4.0
• Epic Color: ghx-label-9
• Assignee: Thomas Tauber-Marshall
• Reporter: Sahil Takiar
• Created: 02/Dec/19 20:08
• Updated: 10/Dec/19 22:29
• Resolved: 10/Dec/19 22:29

Description

custom_cluster.test_rpc_exception.TestRPCException.test_end_data_stream_error (from pytest)

Error Message

assert (not 'Debug Action: IMPALA_SERVICE_POOL:FAIL@0.5' or 8 == 0)  +  where 8 = <bound method TestRPCException._get_num_fails of <test_rpc_exception.TestRPCException object at 0x698c790>>(<ImpaladProcess PID: 5846 (/data/jenkins/workspace/impala-cdh6.x-exhaustive/re..._SERVICE_POOL:127.0.0.1:27002:EndDataStream:FAIL@0.5 --default_query_options=)>)  +    where <bound method TestRPCException._get_num_fails of <test_rpc_exception.TestRPCException object at 0x698c790>> = <test_rpc_exception.TestRPCException object at 0x698c790>._get_num_fails

Stacktrace

custom_cluster/test_rpc_exception.py:123: in test_end_data_stream_error
    self.execute_test_query("Debug Action: IMPALA_SERVICE_POOL:FAIL@0.5")
custom_cluster/test_rpc_exception.py:75: in execute_test_query
    assert not exception_string or self._get_num_fails(impalad) == 0
E   assert (not 'Debug Action: IMPALA_SERVICE_POOL:FAIL@0.5' or 8 == 0)
E    +  where 8 = <bound method TestRPCException._get_num_fails of <test_rpc_exception.TestRPCException object at 0x698c790>>(<ImpaladProcess PID: 5846 (/data/jenkins/workspace/impala-cdh6.x-exhaustive/re..._SERVICE_POOL:127.0.0.1:27002:EndDataStream:FAIL@0.5 --default_query_options=)>)
E    +    where <bound method TestRPCException._get_num_fails of <test_rpc_exception.TestRPCException object at 0x698c790>> = <test_rpc_exception.TestRPCException object at 0x698c790>._get_num_fails

Standard Error

23:14:02 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
23:14:02 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO
23:14:02 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
23:14:02 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO
23:14:02 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
23:14:02 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
23:14:05 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
23:14:05 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
23:14:05 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25000
23:14:05 MainThread: Waiting for num_known_live_backends=3. Current value: 0
23:14:06 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
23:14:06 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25000
23:14:06 MainThread: Waiting for num_known_live_backends=3. Current value: 0
23:14:07 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
23:14:07 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25000
23:14:07 MainThread: num_known_live_backends has reached value: 3
23:14:07 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
23:14:07 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25001
23:14:07 MainThread: num_known_live_backends has reached value: 3
23:14:08 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
23:14:08 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25002
23:14:08 MainThread: num_known_live_backends has reached value: 3
23:14:09 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors).
DEBUG:impala_cluster:Found 3 impalad/1 statestored/1 catalogd process(es)
INFO:impala_service:Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25010
INFO:impala_service:Metric 'statestore.live-backends' has reached desired value: 4
DEBUG:impala_service:Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25000
INFO:impala_service:num_known_live_backends has reached value: 3
DEBUG:impala_service:Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25001
INFO:impala_service:num_known_live_backends has reached value: 3
DEBUG:impala_service:Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-116e.vpc.cloudera.com:25002
INFO:impala_service:num_known_live_backends has reached value: 3