[CASSANDRA-17806] Flaky test_rolling_upgrade

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.28, 3.11.14, 4.0.6, 4.1-beta1, 5.0-alpha1, 5.0
• Component/s: Test/dtest/python
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.1-beta

Source Control Link

https://github.com/apache/cassandra-dtest/commit/24b58342eca597b375a424d08b64f6d001bc220a

Test and Documentation Plan

See PR

• Assignee: Berenguer Blasi
• Reporter: Berenguer Blasi

Authors

Berenguer Blasi

Reviewers

Benjamin Lerer

• Created: 09/Aug/22 06:31
• Updated: 12/Sep/23 13:02
• Resolved: 10/Aug/22 05:15

Description

The fix on CASSANDRA-17140 needs to be extended into other places as it seems it now fails only one in a billion but still we can fix that one.

Regression

dtest-upgrade.upgrade_tests.upgrade_through_versions_test.TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_3_11_X_HEAD.test_rolling_upgrade (from Cassandra dtests)
Failing for the past 1 build (Since
#115 )
Took 10 min.
Failed 1 times in the last 9 runs. Flakiness: 12%, Stability: 88%
Error Message

RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.

Stacktrace

self = <upgrade_tests.upgrade_through_versions_test.TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_3_11_X_HEAD object at 0x7f4d313e4e50>

    @pytest.mark.timeout(3000)
    def test_rolling_upgrade(self):
        """
            Test rolling upgrade of the cluster, so we have mixed versions part way through.
            """
>       self.upgrade_scenario(rolling=True)

upgrade_tests/upgrade_through_versions_test.py:340: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
upgrade_tests/upgrade_through_versions_test.py:417: in upgrade_scenario
    self._check_on_subprocs(self.fixture_dtest_setup.subprocs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <upgrade_tests.upgrade_through_versions_test.TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_3_11_X_HEAD object at 0x7f4d313e4e50>
subprocs = [<Process name='Process-1' pid=10867 parent=389 stopped exitcode=-SIGKILL daemon>, <Process name='Process-2' pid=10881 parent=389 stopped exitcode=1 daemon>]

    def _check_on_subprocs(self, subprocs):
        """
            Check on given subprocesses.
    
            If any are not alive, we'll go ahead and terminate any remaining alive subprocesses since this test is going to fail.
            """
        subproc_statuses = [s.is_alive() for s in subprocs]
        if not all(subproc_statuses):
            message = "A subprocess has terminated early. Subprocess statuses: "
            for s in subprocs:
                message += "{name} (is_alive: {aliveness}), ".format(name=s.name, aliveness=s.is_alive())
            message += "attempting to terminate remaining subprocesses now."
            self._terminate_subprocs()
>           raise RuntimeError(message)
E           RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.

upgrade_tests/upgrade_through_versions_test.py:475: RuntimeError

Issue Links

• is a child of: CASSANDRA-17140 Broken test_rolling_upgrade - upgrade_tests.upgrade_through_versions_test.TestUpgrade_indev_3_0_x_To_indev_4_0_x
• links to: CASSANDRA-17806 PR dtests