[CASSANDRA-5332] 1.2.3 Nodetool reports 1.1.10 nodes as down

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate

Environment

Sun Java 6_u39
Ubuntu 10.04.1
Cassandra 1.2.3 and 1.1.10

• Severity: Normal
• Assignee: Unassigned
• Reporter: Arya Goudarzi
• Created: 11/Mar/13 16:25
• Updated: 16/Apr/19 09:32
• Resolved: 26/Apr/13 21:32

Description

During exercise of a rolling upgrade from 1.1.10 to 1.2.3, I upgraded one node from 1.1.10 to 1.2.3 today. It appears that node running 1.2.3 reports nodes in 1.1.10 as Down in nodetool. This means that the nodes running 1.1.10 see all other nodes including 1.2.3 as Up. Here is the ring and gossip from nodes with 1.1.10 for example. "231.121" is the upgraded node:

Address DC Rack Status State Load Effective-Ownership Token
141784319550391026443072753098378663700
XX.180.36 us-east 1b Up Normal 49.47 GB 25.00% 1808575600
XX.231.121 us-east 1c Up Normal 47.08 GB 25.00% 7089215977519551322153637656637080005
XX.177.177 us-east 1d Up Normal 33.64 GB 25.00% 14178431955039102644307275311465584410
XX.7.148 us-east 1b Up Normal 41.27 GB 25.00% 42535295865117307932921825930779602030
XX.20.9 us-east 1c Up Normal 38.51 GB 25.00% 49624511842636859255075463585608106435
XX.86.255 us-east 1d Up Normal 34.78 GB 25.00% 56713727820156410577229101240436610840
XX.63.230 us-east 1b Up Normal 38.11 GB 25.00% 85070591730234615865843651859750628460
XX.163.36 us-east 1c Up Normal 44.25 GB 25.00% 92159807707754167187997289514579132865
XX.31.234 us-east 1d Up Normal 44.66 GB 25.00% 99249023685273718510150927169407637270
XX.132.169 us-east 1b Up Normal 44.2 GB 25.00% 127605887595351923798765477788721654890
XX.71.63 us-east 1c Up Normal 38.74 GB 25.00% 134695103572871475120919115443550159295
XX.197.209 us-east 1d Up Normal 41.5 GB 25.00% 141784319550391026443072753098378663700

/XX.71.63
RACK:1c
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.1598705272E10
DC:us-east
INTERNAL_IP:XX.194.92
STATUS:NORMAL,134695103572871475120919115443550159295
RPC_ADDRESS:XX.194.92
RELEASE_VERSION:1.1.6
/XX.86.255
RACK:1d
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:3.734334162E10
DC:us-east
INTERNAL_IP:XX.6.195
STATUS:NORMAL,56713727820156410577229101240436610840
RPC_ADDRESS:XX.6.195
RELEASE_VERSION:1.1.6
/XX.7.148
RACK:1b
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.4316975808E10
DC:us-east
INTERNAL_IP:XX.47.250
STATUS:NORMAL,42535295865117307932921825930779602030
RPC_ADDRESS:XX.47.250
RELEASE_VERSION:1.1.6
/XX.63.230
RACK:1b
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.0918593305E10
DC:us-east
INTERNAL_IP:XX.89.127
STATUS:NORMAL,85070591730234615865843651859750628460
RPC_ADDRESS:XX.89.127
RELEASE_VERSION:1.1.6
/XX.132.169
RACK:1b
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.745883458E10
DC:us-east
INTERNAL_IP:XX.94.161
STATUS:NORMAL,127605887595351923798765477788721654890
RPC_ADDRESS:XX.94.161
RELEASE_VERSION:1.1.6
/XX.180.36
RACK:1b
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:5.311963027E10
DC:us-east
INTERNAL_IP:XX.123.112
STATUS:NORMAL,1808575600
RPC_ADDRESS:XX.123.112
RELEASE_VERSION:1.1.6
/XX.163.36
RACK:1c
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.7516755022E10
DC:us-east
INTERNAL_IP:XX.163.180
STATUS:NORMAL,92159807707754167187997289514579132865
RPC_ADDRESS:XX.163.180
RELEASE_VERSION:1.1.6
/XX.31.234
RACK:1d
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.7954372912E10
DC:us-east
INTERNAL_IP:XX.192.159
STATUS:NORMAL,99249023685273718510150927169407637270
RPC_ADDRESS:XX.192.159
RELEASE_VERSION:1.1.6
/XX.197.209
RACK:1d
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.4558968005E10
DC:us-east
INTERNAL_IP:XX.66.205
STATUS:NORMAL,141784319550391026443072753098378663700
RPC_ADDRESS:XX.66.205
RELEASE_VERSION:1.1.6
/XX.177.177
RACK:1d
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:3.6115572697E10
DC:us-east
INTERNAL_IP:XX.65.57
STATUS:NORMAL,14178431955039102644307275311465584410
RPC_ADDRESS:XX.65.57
RELEASE_VERSION:1.1.6
/XX.20.9
RACK:1c
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
LOAD:4.1352503882E10
DC:us-east
INTERNAL_IP:XX.33.229
STATUS:NORMAL,49624511842636859255075463585608106435
RPC_ADDRESS:XX.33.229
RELEASE_VERSION:1.1.6
/XX.231.121
RACK:1c
SCHEMA:09487aa5-3380-33ab-b9a5-bcc8476066b0
X4:9c765678-d058-4d85-a588-638ce10ff984
X3:7
DC:us-east
INTERNAL_IP:XX.223.241
RPC_ADDRESS:XX.223.241
RELEASE_VERSION:1.2.2

Now the nodetool on the 1.2.3 node shows all nodes as Down but itself. Gossipinfo looks good though:

Datacenter: us-east
==========
Replicas: 3

Address Rack Status State Load Owns Token
56713727820156410577229101240436610840
XX.132.169 1b Down Normal 44.2 GB 25.00% 127605887595351923798765477788721654890
XX.7.148 1b Down Normal 41.27 GB 25.00% 42535295865117307932921825930779602030
XX.180.36 1b Down Normal 49.47 GB 25.00% 1808575600
XX.63.230 1b Down Normal 38.11 GB 25.00% 85070591730234615865843651859750628460
XX.231.121 1c Up Normal 47.25 GB 25.00% 7089215977519551322153637656637080005
XX.71.63 1c Down Normal 38.74 GB 25.00% 134695103572871475120919115443550159295
XX.177.177 1d Down Normal 33.64 GB 25.00% 14178431955039102644307275311465584410
XX.31.234 1d Down Normal 44.66 GB 25.00% 99249023685273718510150927169407637270
XX.20.9 1c Down Normal 38.51 GB 25.00% 49624511842636859255075463585608106435
XX.163.36 1c Down Normal 44.25 GB 25.00% 92159807707754167187997289514579132865
XX.197.209 1d Down Normal 41.5 GB 25.00% 141784319550391026443072753098378663700
XX.86.255 1d Down Normal 34.78 GB 25.00% 56713727820156410577229101240436610840

/XX.71.63
RACK:1c
RPC_ADDRESS:XX.194.92
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.194.92
STATUS:NORMAL,134695103572871475120919115443550159295
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.1598705272E10
/XX.86.255
RACK:1d
RPC_ADDRESS:XX.6.195
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.6.195
STATUS:NORMAL,56713727820156410577229101240436610840
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:3.7343205002E10
/XX.7.148
RACK:1b
RPC_ADDRESS:XX.47.250
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.47.250
STATUS:NORMAL,42535295865117307932921825930779602030
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.4316975808E10
/XX.63.230
RACK:1b
RPC_ADDRESS:XX.89.127
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.89.127
STATUS:NORMAL,85070591730234615865843651859750628460
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.0918456687E10
/XX.132.169
RACK:1b
RPC_ADDRESS:XX.94.161
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.94.161
STATUS:NORMAL,127605887595351923798765477788721654890
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.745883458E10
/XX.180.36
RACK:1b
RPC_ADDRESS:XX.123.112
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.123.112
STATUS:NORMAL,1808575600
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:5.311963027E10
/XX.163.36
RACK:1c
RPC_ADDRESS:XX.163.180
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.163.180
STATUS:NORMAL,92159807707754167187997289514579132865
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.7516755022E10
/XX.31.234
RACK:1d
RPC_ADDRESS:XX.192.159
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.192.159
STATUS:NORMAL,99249023685273718510150927169407637270
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.7954372912E10
/XX.197.209
RACK:1d
RPC_ADDRESS:XX.66.205
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.66.205
STATUS:NORMAL,141784319550391026443072753098378663700
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.4559013211E10
/XX.177.177
RACK:1d
RPC_ADDRESS:XX.65.57
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.65.57
STATUS:NORMAL,14178431955039102644307275311465584410
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:3.6115572697E10
/XX.20.9
RACK:1c
RPC_ADDRESS:XX.33.229
RELEASE_VERSION:1.1.6
INTERNAL_IP:XX.33.229
STATUS:NORMAL,49624511842636859255075463585608106435
SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
DC:us-east
LOAD:4.1352367264E10
/XX.231.121
HOST_ID:9c765678-d058-4d85-a588-638ce10ff984
RACK:1c
RPC_ADDRESS:XX.223.241
RELEASE_VERSION:1.2.2
INTERNAL_IP:XX.223.241
STATUS:NORMAL,7089215977519551322153637656637080005
NET_VERSION:7
SCHEMA:8b8948f5-d56f-3a96-8005-b9452e42cd67
SEVERITY:0.0
DC:us-east
LOAD:5.0710624207E10

Is this just a display bug in nodetool or this upgraded node really sees the other ones as dead?

Issue Links

• duplicates: CASSANDRA-5432 Repair Freeze/Gossip Invisibility Issues 1.2.4