Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16334

Replica failure causes timeout on multi-DC write

    XMLWordPrintableJSON

    Details

    • Bug Category:
      Correctness - API / Semantic Implementation
    • Severity:
      Normal
    • Complexity:
      Normal
    • Discovered By:
      User Report
    • Platform:
      All
    • Impacts:
      None

      Description

      Inserting a mutation larger than max_mutation_size_in_kb correctly throws a write error on a single DC keyspace with RF=3:

      cassandra.WriteFailure: Error from server: code=1500 [Replica(s) failed to execute write] message="Operation failed - received 0 responses and 3 failures: UNKNOWN from /127.0.0.3:7000, UNKNOWN from /127.0.0.2:7000, UNKNOWN from /127.0.0.1:7000" info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0, 'failures': 3}
      

      The same insert wrongly causes a timeout on a keyspace with 2 dcs (RF=3 each):

      cassandra.WriteTimeout: Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0}
      

      Reproduction steps:

      # Setup cluster
      ccm create -n 3:3 test
      for i in {1..6}; do echo 'max_mutation_size_in_kb: 1000' >> ~/.ccm/test/node$i/conf/cassandra.yaml; done
      ccm start
      
      # Create schema
      ccm node1 cqlsh
      CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};
      CREATE TABLE test.test (key int PRIMARY KEY, val blob);
      exit;
      
      # Insert data
      python
      from cassandra.cluster import Cluster
      session = cluster.connect('test')
      blob = f = open("2mbBlob", "rb").read().hex()
      session.execute("INSERT INTO test (key, val) VALUES (1, textAsBlob('" + blob + "'))")
      

      Reproduced in 3.11, trunk.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              paulo Paulo Motta
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: