[CASSANDRA-11170] Uneven load can be created by cross DC mutation propagations, as remote forwarding node is not randomly picked

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Component/s: Legacy/Coordination
• Assignee: Wei Deng
• Reporter: Wei Deng

Authors

Wei Deng

• Created: 13/Feb/16 10:15
• Updated: 18/Jul/22 21:20
• Resolved: 18/Jul/22 21:20

Description

I was looking at the o.a.c.service.StorageProxy code and realized that it seems to be always picking the first IP in the remote DC target list as the destination, whenever it needs to send the mutation to a remote DC. See these lines in the code:

https://github.com/apache/cassandra/blob/1944bf507d66b5c103c136319caeb4a9e3767a69/src/java/org/apache/cassandra/service/StorageProxy.java#L1280-L1301

This could cause one node in the remote DC receiving more mutation messages than the other nodes, and hence uneven workload distribution.

A trivial test (with TRACE logging level enabled) on a 3+3 node cluster proved the problem, see the system.log entries below:

INFO  [RMI TCP Connection(18)-54.173.227.52] 2016-02-13 09:54:55,948  StorageService.java:3353 - set log level to TRACE for classes under 'org.apache.cassandra.service.StorageProxy' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE')
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:15,148  StorageProxy.java:1284 - Adding FWD message to 8996@/52.53.215.74
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:15,149  StorageProxy.java:1284 - Adding FWD message to 8997@/54.183.23.201
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:15,149  StorageProxy.java:1289 - Sending message to 8998@/54.183.209.219
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:22,939  StorageProxy.java:1284 - Adding FWD message to 9032@/52.53.215.74
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:22,940  StorageProxy.java:1284 - Adding FWD message to 9033@/54.183.23.201
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:22,941  StorageProxy.java:1289 - Sending message to 9034@/54.183.209.219
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:28,975  StorageProxy.java:1284 - Adding FWD message to 9064@/52.53.215.74
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:28,976  StorageProxy.java:1284 - Adding FWD message to 9065@/54.183.23.201
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:28,977  StorageProxy.java:1289 - Sending message to 9066@/54.183.209.219
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:33,464  StorageProxy.java:1284 - Adding FWD message to 9094@/52.53.215.74
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:33,465  StorageProxy.java:1284 - Adding FWD message to 9095@/54.183.23.201
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:33,478  StorageProxy.java:1289 - Sending message to 9096@/54.183.209.219
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:39,243  StorageProxy.java:1284 - Adding FWD message to 9121@/52.53.215.74
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:39,244  StorageProxy.java:1284 - Adding FWD message to 9122@/54.183.23.201
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:39,244  StorageProxy.java:1289 - Sending message to 9123@/54.183.209.219
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:44,248  StorageProxy.java:1284 - Adding FWD message to 9145@/52.53.215.74
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:44,249  StorageProxy.java:1284 - Adding FWD message to 9146@/54.183.23.201
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:44,249  StorageProxy.java:1289 - Sending message to 9147@/54.183.209.219
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:49,731  StorageProxy.java:1284 - Adding FWD message to 9170@/52.53.215.74
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:49,734  StorageProxy.java:1284 - Adding FWD message to 9171@/54.183.23.201
TRACE [SharedPool-Worker-1] 2016-02-13 09:55:49,735  StorageProxy.java:1289 - Sending message to 9172@/54.183.209.219
INFO  [RMI TCP Connection(22)-54.173.227.52] 2016-02-13 09:56:19,545  StorageService.java:3353 - set log level to INFO for classes under 'org.apache.cassandra.service.StorageProxy' (if the level doesn't look like 'INFO' then the logger couldn't parse 'INFO')

Issue Links

• is fixed by: CASSANDRA-15318 sendMessagesToNonlocalDC() should shuffle targets