Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16334

Replica failure causes timeout on multi-DC write

    XMLWordPrintableJSON

Details

    • Correctness - API / Semantic Implementation
    • Normal
    • Normal
    • User Report
    • All
    • None
    • NA
    • Hide

      Added a dtest that covers both bugs causing replica failures to appear as timeouts.

      Show
      Added a dtest that covers both bugs causing replica failures to appear as timeouts.

    Description

      Inserting a mutation larger than max_mutation_size_in_kb correctly throws a write error on a single DC keyspace with RF=3:

      cassandra.WriteFailure: Error from server: code=1500 [Replica(s) failed to execute write] message="Operation failed - received 0 responses and 3 failures: UNKNOWN from /127.0.0.3:7000, UNKNOWN from /127.0.0.2:7000, UNKNOWN from /127.0.0.1:7000" info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0, 'failures': 3}
      

      The same insert wrongly causes a timeout on a keyspace with 2 dcs (RF=3 each):

      cassandra.WriteTimeout: Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0}
      

      Reproduction steps:

      # Setup cluster
      ccm create -n 3:3 test
      for i in {1..6}; do echo 'max_mutation_size_in_kb: 1000' >> ~/.ccm/test/node$i/conf/cassandra.yaml; done
      ccm start
      
      # Create schema
      ccm node1 cqlsh
      CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};
      CREATE TABLE test.test (key int PRIMARY KEY, val blob);
      exit;
      
      # Insert data
      python
      from cassandra.cluster import Cluster
      cluster = Cluster()
      session = cluster.connect('test')
      blob = f = open("2mbBlob", "rb").read().hex()
      session.execute("INSERT INTO test (key, val) VALUES (1, textAsBlob('" + blob + "'))")
      

      Reproduced in 3.0, 3.11, 4.0, trunk.

      Attachments

        Issue Links

          Activity

            People

              Gerrrr Aleksandr Sorokoumov
              paulo Paulo Motta
              Aleksandr Sorokoumov
              Andres de la Peña, Paulo Motta
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h