[CASSANDRA-14429] Flaky dTest: bootstrap_test.TestBootstrap.test_local_quorum_bootstrap

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Auto Closed
• Component/s: Test/dtest/python
• Labels: Testing dtest
• Severity: Normal
• Assignee: Unassigned
• Reporter: Jay Zhuang
• Created: 30/Apr/18 18:27
• Updated: 13/Sep/21 17:12
• Resolved: 13/Sep/21 17:12

Description

https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-dtest/lastCompletedBuild/testReport/bootstrap_test/TestBootstrap/test_local_quorum_bootstrap/

Failed 13 times in the last 30 runs. Flakiness: 41%, Stability: 56%

Error Message
Failed: Timeout >900.0s
Stacktrace
self = <bootstrap_test.TestBootstrap object at 0x7fb3c19c2588>

    def test_local_quorum_bootstrap(self):
        """
            Test that CL local_quorum works while a node is bootstrapping.
            @jira_ticket CASSANDRA-8058
            """
        cluster = self.cluster
        cluster.populate([1, 1])
        cluster.start()
    
        node1 = cluster.nodes['node1']
        yaml_config = """
            # Create the keyspace and table
            keyspace: keyspace1
            keyspace_definition: |
              CREATE KEYSPACE keyspace1 WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 1, 'dc2': 1};
            table: users
            table_definition:
              CREATE TABLE users (
                username text,
                first_name text,
                last_name text,
                email text,
                PRIMARY KEY(username)
              ) WITH compaction = {'class':'SizeTieredCompactionStrategy'};
            insert:
              partitions: fixed(1)
              batchtype: UNLOGGED
            queries:
              read:
                cql: select * from users where username = ?
                fields: samerow
            """
        with tempfile.NamedTemporaryFile(mode='w+') as stress_config:
            stress_config.write(yaml_config)
            stress_config.flush()
            node1.stress(['user', 'profile=' + stress_config.name, 'n=2M', 'no-warmup',
>                         'ops(insert=1)', '-rate', 'threads=50'])

bootstrap_test.py:439: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../venv/src/ccm/ccmlib/node.py:1339: in stress
    return handle_external_tool_process(p, ['stress'] + stress_options)
../venv/src/ccm/ccmlib/node.py:2101: in handle_external_tool_process
    out, err = process.communicate()
/usr/lib/python3.6/subprocess.py:843: in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
/usr/lib/python3.6/subprocess.py:1514: in _communicate
    ready = selector.select(timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <selectors.PollSelector object at 0x7fb3c01923c8>, timeout = None

    def select(self, timeout=None):
        if timeout is None:
            timeout = None
        elif timeout <= 0:
            timeout = 0
        else:
            # poll() has a resolution of 1 millisecond, round away from
            # zero to wait *at least* timeout seconds.
            timeout = math.ceil(timeout * 1e3)
        ready = []
        try:
>           fd_event_list = self._poll.poll(timeout)
E           Failed: Timeout >900.0s

/usr/lib/python3.6/selectors.py:376: Failed