[IMPALA-3693] breakpad tests generate core files that get wrongly earmarked for collection

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0, Impala 2.7.0
• Fix Version/s: Impala 2.6.0
• Component/s: Infrastructure
• Target Version: Impala 2.6.0, Impala 2.7.0
• Assignee: Lars Volker
• Reporter: Michael Brown
• Created: 07/Jun/16 19:42
• Updated: 09/Jun/16 10:56
• Resolved: 09/Jun/16 10:56

Description

When the breakpad tests actually run in Jenkins (IMPALA-3630), they generate core files. Internally to Cloudera we collect any core files we find for analysis/collection, because we expect need to analyze them later.

For breakpad tests that rightly generate cores, though, we don't need those cores. Collecting these cores will add time and storage to each build, and also extra info in the saved backtraces.txt file.

http://sandbox.jenkins.cloudera.com/job/impala-umbrella-build-and-test/1520/

+ /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala-auxiliary-tests/jenkins/archive_core_dumps.py --search_dir=/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala --archive_dir=/data1 --archive_file_name=cores-impala-umbrella-build-and-test-1520.tar.gz
2016-06-07 12:12:05,829 - archive_core_dumps - INFO - Found core files: ['./tests/core.1465323860.25165.statestored', './tests/core.1465323860.25172.catalogd', './tests/core.1465323861.25229.impalad', './tests/core.1465323861.25265.impalad', './tests/core.1465323861.25318.impalad', './tests/core.1465323886.26099.statestored', './tests/core.1465323886.26106.catalogd', './tests/core.1465323887.26163.impalad', './tests/core.1465323887.26200.impalad', './tests/core.1465323887.26250.impalad', './tests/core.1465323998.27067.statestored', './tests/core.1465323999.27074.catalogd', './tests/core.1465323999.27131.impalad', './tests/core.1465323999.27167.impalad', './tests/core.1465324000.27220.impalad', './tests/core.1465324152.28100.impalad', './tests/core.1465324152.28132.impalad', './tests/core.1465324152.28173.impalad', './tests/core.1465324152.28035.statestored', './tests/core.1465324153.28042.catalogd', './tests/core.1465324309.29795.impalad']
2016-06-07 12:12:17,841 - archive_core_dumps - INFO - Found binary files: ['./be/build/latest/statestore/statestored', './be/build/latest/catalog/catalogd', './be/build/latest/service/impalad', './be/build/latest/service/impalad', './be/build/latest/service/impalad', './be/build/latest/statestore/statestored', './be/build/latest/catalog/catalogd', './be/build/latest/service/impalad', './be/build/latest/service/impalad', './be/build/latest/service/impalad', './be/build/latest/statestore/statestored', './be/build/latest/catalog/catalogd', './be/build/latest/service/impalad', './be/build/latest/service/impalad', './be/build/latest/service/impalad', './be/build/latest/service/impalad', './be/build/latest/service/impalad', './be/build/latest/service/impalad', './be/build/latest/statestore/statestored', './be/build/latest/catalog/catalogd', './be/build/latest/service/impalad']
2016-06-07 12:12:18,403 - archive_core_dumps - INFO - Found latest impalad binary: ./be/build/latest/service/impalad

It seems like a simple solution would be to use sysctl to cache the current value of kernel.core_pattern, set it to something else that includes a location where we don't look for cores, like /tmp and then re-set it again. This would need to work even if tests fail so that tests down the line can still write cores to the correct location.

Issue Links

• is related to: IMPALA-3684 breakpad tests fail when enabled and run in Jenkins