[CASSANDRA-17085] commit log was switched from non-daemon to daemon threads, which causes the JVM to exit in some case as no non-daemon threads are active

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1-alpha1, 4.1
• Component/s: Test/dtest/python
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra/commit/a540afd12ae4c29f15c54330242efdf1cf61354c

Test and Documentation Plan

local and ci dtest

• Assignee: David Capwell
• Reporter: David Capwell

Authors

David Capwell, Sam Tunnicliffe

Reviewers

Sam Tunnicliffe

• Created: 29/Oct/21 19:08
• Updated: 27/May/22 19:25
• Resolved: 03/Nov/21 20:08

Description

Right now bootstrap tests are failing every time we run, this work is to debug and fix the underling issue.

Examples:

https://app.circleci.com/pipelines/github/dcapwell/cassandra/1062/workflows/ba3e6395-ef22-4724-8424-0549e65d8cff/jobs/7089

>       node3.nodetool('bootstrap resume')

bootstrap_test.py:1014: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../env3.6/lib/python3.6/site-packages/ccmlib/node.py:1005: in nodetool
    return handle_external_tool_process(p, ['nodetool', '-h', 'localhost', '-p', str(self.jmx_port)] + shlex.split(cmd))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

process = <subprocess.Popen object at 0x7fb071a03940>
cmd_args = ['nodetool', '-h', 'localhost', '-p', '7300', 'bootstrap', ...]

    def handle_external_tool_process(process, cmd_args):
        out, err = process.communicate()
        if (out is not None) and isinstance(out, bytes):
            out = out.decode()
        if (err is not None) and isinstance(err, bytes):
            err = err.decode()
        rc = process.returncode
    
        if rc != 0:
>           raise ToolError(cmd_args, rc, out, err)
E           ccmlib.node.ToolError: Subprocess ['nodetool', '-h', 'localhost', '-p', '7300', 'bootstrap', 'resume'] exited with non-zero status; exit status: 1; 
E           stderr: nodetool: Failed to connect to 'localhost:7300' - EOFException: 'null'.

../env3.6/lib/python3.6/site-packages/ccmlib/node.py:2305: ToolError

https://app.circleci.com/pipelines/github/dcapwell/cassandra/1062/workflows/ba3e6395-ef22-4724-8424-0549e65d8cff/jobs/7087

>       node1.start()

bootstrap_test.py:483: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../env3.6/lib/python3.6/site-packages/ccmlib/node.py:895: in start
    node.watch_log_for_alive(self, from_mark=mark)
../env3.6/lib/python3.6/site-packages/ccmlib/node.py:664: in watch_log_for_alive
    self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, filename=filename)
../env3.6/lib/python3.6/site-packages/ccmlib/node.py:592: in watch_log_for
    head=reads[:50], tail="..."+reads[len(reads)-150:]))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

start = 1635453190.3118386, timeout = 120
msg = "Missing: ['127.0.0.1:7000.* is now UP'] not found in system.log:\n Head: \n Tail: ..."
node = 'node3'

    @staticmethod
    def raise_if_passed(start, timeout, msg, node=None):
        if start + timeout < time.time():
>           raise TimeoutError.create(start, timeout, msg, node)
E           ccmlib.node.TimeoutError: 28 Oct 2021 20:35:10 [node3] after 120.12/120 seconds Missing: ['127.0.0.1:7000.* is now UP'] not found in system.log:
E            Head: 
E            Tail: ...

Issue Links

• duplicates: CASSANDRA-17081 Fix test: bootstrap_test.py::TestBootstrap::test_bootstrap_with_reset_bootstrap_state
• relates to: CASSANDRA-17116 When streaming sees a ClosedChannelException this triggers the disk failure policy
• links to: Circle CI: trunk
• links to: GH PR: trunk