[IMPALA-6023] impalad failed to start in test_dcheck_writes_minidump

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Labels: broken-build
• Epic Color: ghx-label-5
• Assignee: Lars Volker
• Reporter: Michael Brown
• Created: 06/Oct/17 17:20
• Updated: 13/Oct/17 01:54
• Resolved: 13/Oct/17 01:53

Description

lv can you take a look?

self = <test_breakpad.TestBreakpad object at 0x5c83290>

    @SkipIfBuildType.not_dev_build
    @pytest.mark.execute_serially
    def test_dcheck_writes_minidump(self):
      """Check that hitting a DCHECK macro writes a minidump."""
      assert self.count_all_minidumps() == 0
      failed_to_start = False
      try:
        self.start_cluster_with_args(minidump_path=self.tmp_dir,
            beeswax_port=1)
      except CalledProcessError:
        failed_to_start = True
      assert failed_to_start
>     assert self.count_minidumps('impalad') > 0
E     assert 0 > 0
E      +  where 0 = <bound method TestBreakpad.count_minidumps of <test_breakpad.TestBreakpad object at 0x5c83290>>('impalad')
E      +    where <bound method TestBreakpad.count_minidumps of <test_breakpad.TestBreakpad object at 0x5c83290>> = <test_breakpad.TestBreakpad object at 0x5c83290>.count_minidumps

custom_cluster/test_breakpad.py:321: AssertionError
----------------------------- Captured stdout call -----------------------------
Starting State Store logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/statestored.INFO
Starting Catalog Service logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
Starting Impala Daemon logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad.INFO
Starting Impala Daemon logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
Starting Impala Daemon logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
Error starting cluster: num_known_live_backends did not reach expected value in time
----------------------------- Captured stderr call -----------------------------
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-0055.vpc.cloudera.com:25000
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
...
MainThread: Debug webpage did not become available in expected time.
MainThread: Waiting for num_known_live_backends=3. Current value: None
MainThread: Found 0 impalad/1 statestored/1 catalogd process(es)

This was the only test to fail this way:

16:36:32 custom_cluster/test_alloc_fail.py .x
16:44:17 custom_cluster/test_breakpad.py .........F
16:44:44 custom_cluster/test_client_ssl.py .sss

Issue Links

• breaks: IMPALA-6049 Fix for IMPALA-6023 breaks test_breakpad on localFs builds
• is broken by: IMPALA-4786 Refactor CreateImpalaServer() to allow it to be used in tests.