[CASSANDRA-8999] Number of failing dtests running locally is different from running in Jenkins

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Component/s: CI
• Labels: CI monthly-release
• Severity: Normal
• Assignee: Tom Hobbs
• Reporter: Ariel Weisberg

Authors

Tom Hobbs

• Created: 19/Mar/15 21:24
• Updated: 13/May/20 13:18
• Resolved: 14/Aug/15 15:54

Description

It's desirable that Jenkin's and running locally give as close to the same answer as possible. It could be a red herring and something about my environment. As a starting point run the tests locally and check if the result is the same as when I did it.
In Jenkin's there were 150 failures: http://cassci.datastax.com/job/CTOOL_trunk_dtest/9/
When I ran locally I got the following failures:

aweisberg@aweisberg-XH61:~/git/cstar/cassandra-dtest$ ls | grep ".log" | xargs grep "ERROR:"
commitlog_test.log:ERROR: default_compressed_segment_size_test (commitlog_test.TestCommitLog)
commitlog_test.log:ERROR: small_compressed_segment_size_test (commitlog_test.TestCommitLog)
global_row_key_cache_test.log:ERROR: functional_test (global_row_key_cache_test.TestGlobalRowKeyCache)
replace_address_test.log:ERROR: replace_first_boot_test (replace_address_test.TestReplaceAddress)
replace_address_test.log:ERROR: replace_first_boot_test (replace_address_test.TestReplaceAddress)
secondary_indexes_test.log:ERROR: test_low_cardinality_indexes (secondary_indexes_test.TestSecondaryIndexes)
super_column_cache_test.log:ERROR: sc_with_row_cache_test (super_column_cache_test.TestSCCache)
super_column_cache_test.log:cassandra.cluster: ERROR: Control connection failed to connect, shutting down Cluster:
thrift_hsha_test.log:ERROR: Failure: ImportError (No module named pycassa)
upgrade_supercolumns_test.log:ERROR: upgrade_with_counters_test (upgrade_supercolumns_test.TestSCUpgrade)
upgrade_supercolumns_test.log:ERROR: upgrade_with_index_creation_test (upgrade_supercolumns_test.TestSCUpgrade)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestRandomPartitionerUpgrade)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestRandomPartitionerUpgrade)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestUpgradeThroughVersions)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgradeThroughVersions)
upgrade_through_versions_test.log:ERROR: bootstrap_multidc_test (upgrade_through_versions_test.TestUpgrade_from_1_2_latest_tag_to_1_2_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_test (upgrade_through_versions_test.TestUpgrade_from_1_2_latest_tag_to_1_2_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestUpgrade_from_1_2_latest_tag_to_1_2_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_1_2_latest_tag_to_1_2_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_multidc_test (upgrade_through_versions_test.TestUpgrade_from_1_2_latest_tag_to_cassandra_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_test (upgrade_through_versions_test.TestUpgrade_from_1_2_latest_tag_to_cassandra_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestUpgrade_from_1_2_latest_tag_to_cassandra_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_1_2_latest_tag_to_cassandra_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_multidc_test (upgrade_through_versions_test.TestUpgrade_from_2_0_latest_tag_to_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_test (upgrade_through_versions_test.TestUpgrade_from_2_0_latest_tag_to_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestUpgrade_from_2_0_latest_tag_to_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_2_0_latest_tag_to_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_multidc_test (upgrade_through_versions_test.TestUpgrade_from_2_0_latest_tag_to_cassandra_2_1_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_test (upgrade_through_versions_test.TestUpgrade_from_2_0_latest_tag_to_cassandra_2_1_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestUpgrade_from_2_0_latest_tag_to_cassandra_2_1_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_2_0_latest_tag_to_cassandra_2_1_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_trunk_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_trunk_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_multidc_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_1_2_HEAD_to_cassandra_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_1_2_HEAD_to_cassandra_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_1_2_HEAD_to_cassandra_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_cassandra_1_2_HEAD_to_cassandra_2_0_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_multidc_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_1_2_HEAD_to_cassandra_2_0_latest_tag)
upgrade_through_versions_test.log:ERROR: bootstrap_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_1_2_HEAD_to_cassandra_2_0_latest_tag)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_1_2_HEAD_to_cassandra_2_0_latest_tag)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_cassandra_1_2_HEAD_to_cassandra_2_0_latest_tag)
upgrade_through_versions_test.log:ERROR: bootstrap_multidc_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_0_HEAD_to_cassandra_2_1_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_0_HEAD_to_cassandra_2_1_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_0_HEAD_to_cassandra_2_1_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_0_HEAD_to_cassandra_2_1_HEAD)
upgrade_through_versions_test.log:ERROR: bootstrap_multidc_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_0_HEAD_to_cassandra_2_1_latest_tag)
upgrade_through_versions_test.log:ERROR: bootstrap_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_0_HEAD_to_cassandra_2_1_latest_tag)
upgrade_through_versions_test.log:ERROR: upgrade_test (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_0_HEAD_to_cassandra_2_1_latest_tag)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_0_HEAD_to_cassandra_2_1_latest_tag)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_1_HEAD_to_trunk_HEAD)
upgrade_through_versions_test.log:ERROR: upgrade_test_mixed (upgrade_through_versions_test.TestUpgrade_from_cassandra_2_1_HEAD_to_trunk_HEAD)
user_functions_test.log:ERROR: test_migration (user_functions_test.TestUserFunctions)
user_functions_test.log:ERROR: udf_overload_test (user_functions_test.TestUserFunctions)
user_functions_test.log:ERROR: udf_overload_test (user_functions_test.TestUserFunctions)
user_functions_test.log:ERROR: udf_scripting_test (user_functions_test.TestUserFunctions)
aweisberg@aweisberg-XH61:~/git/cstar/cassandra-dtest$ ls | grep ".log" | xargs grep "ERROR:" | wc -l
55

Issue Links

• is depended upon by: CASSANDRA-8996 dtests should pass on trunk