Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16334

Replica failure causes timeout on multi-DC write

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Correctness - API / Semantic Implementation
    • Normal
    • Normal
    • User Report
    • All
    • None
    • NA
    • Hide

      Added a dtest that covers both bugs causing replica failures to appear as timeouts.

      Show
      Added a dtest that covers both bugs causing replica failures to appear as timeouts.

    Description

      Inserting a mutation larger than max_mutation_size_in_kb correctly throws a write error on a single DC keyspace with RF=3:

      cassandra.WriteFailure: Error from server: code=1500 [Replica(s) failed to execute write] message="Operation failed - received 0 responses and 3 failures: UNKNOWN from /127.0.0.3:7000, UNKNOWN from /127.0.0.2:7000, UNKNOWN from /127.0.0.1:7000" info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0, 'failures': 3}
      

      The same insert wrongly causes a timeout on a keyspace with 2 dcs (RF=3 each):

      cassandra.WriteTimeout: Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0}
      

      Reproduction steps:

      # Setup cluster
      ccm create -n 3:3 test
      for i in {1..6}; do echo 'max_mutation_size_in_kb: 1000' >> ~/.ccm/test/node$i/conf/cassandra.yaml; done
      ccm start
      
      # Create schema
      ccm node1 cqlsh
      CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};
      CREATE TABLE test.test (key int PRIMARY KEY, val blob);
      exit;
      
      # Insert data
      python
      from cassandra.cluster import Cluster
      cluster = Cluster()
      session = cluster.connect('test')
      blob = f = open("2mbBlob", "rb").read().hex()
      session.execute("INSERT INTO test (key, val) VALUES (1, textAsBlob('" + blob + "'))")
      

      Reproduced in 3.0, 3.11, 4.0, trunk.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Gerrrr Aleksandr Sorokoumov Assign to me
            paulo Paulo Motta
            Aleksandr Sorokoumov
            Andres de la Peña, Paulo Motta
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h
                1h

                Issue deployment