[IMPALA-3794] test_breakpad.py is flaky

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0, Impala 2.8.0, Impala 2.9.0
• Fix Version/s: Impala 2.9.0
• Component/s: Infrastructure
• Labels: breakpad broken-build flaky
• Assignee: Lars Volker
• Reporter: Lars Volker
• Created: 27/Jun/16 22:42
• Updated: 09/Apr/17 10:36
• Resolved: 09/Apr/17 10:36

Description

Using the following command usually fails within an hour: while [ $? -eq 0 ]; do ./run-tests.py ./custom_cluster/test_breakpad.py --exploration_strategy=exhaustive -k test_minidump_relative_path -x -s; done. The problem is, that only two of the three impalad processes will write a minidump, while one of them won't. However in that case there is no breakpad related error message in its logfile.

Creating the minidump_base_dir seems to alleviate the problem, suggesting there might be a race condition somewhere in the call to boost::filesystem::create_directories.

This problem only seems to arise in the specific scenario of multiple impalad processes sharing the same minidump_base_dir which also has to be non-existent. Outside of tests it seems like it cannot occur.

Issue Links

• relates to: IMPALA-5187 Bump breakpad version to include the fix for Breakpad #681, re-enable the strict check that was disabled in IMPALA-3794