Details
-
Bug
-
Status: Resolved
-
Urgent
-
Resolution: Invalid
-
None
-
None
-
This is for Cassandra 2.1.13. At times there are replication delays across multiple regions. Data is available (getting queried from command line) in 1 region but not seen in other region(s). This is not consistent. It is cluster spanning multiple data centers with total > 30 nodes. Keyspace is configured to get replicated in all the data centers.
Hints are getting piled up in the source region. This happens especially for large data payload (appro 1kb to few MB blobs). Network level congestion or saturation does not seem to be an issue. There is no memory/cpu pressure on individual nodes.
I am attaching Cassandra.yaml, any pointers on what can be tuned are highly appreciated. Let me know if you need any other info.
We tried bumping up hinted_handoff_throttle_in_kb: 30720 and handoff tends to be slower max_hints_delivery_threads: 12 on one of the nodes to see if it speeds up hints delivery, there was some improvement but not whole lot.
This is for Cassandra 2.1.13. At times there are replication delays across multiple regions. Data is available (getting queried from command line) in 1 region but not seen in other region(s). This is not consistent. It is cluster spanning multiple data centers with total > 30 nodes. Keyspace is configured to get replicated in all the data centers. Hints are getting piled up in the source region. This happens especially for large data payload (appro 1kb to few MB blobs). Network level congestion or saturation does not seem to be an issue. There is no memory/cpu pressure on individual nodes. I am attaching Cassandra.yaml, any pointers on what can be tuned are highly appreciated. Let me know if you need any other info. We tried bumping up hinted_handoff_throttle_in_kb: 30720 and handoff tends to be slower max_hints_delivery_threads: 12 on one of the nodes to see if it speeds up hints delivery, there was some improvement but not whole lot.
-
Critical