[CASSANDRA-5190] peers listed in peer system table but not listed in ring or gossipinfo

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Duplicate

Environment

Ubuntu 12.04

• Severity: Critical
• Assignee: Unassigned
• Reporter: Michael Kjellman
• Created: 28/Jan/13 04:01
• Updated: 16/Apr/19 09:32
• Resolved: 28/Jan/13 18:23

Description

cqlsh:system> select * from peers;         

 peer        | data_center | host_id                              | rack | release_version | rpc_address | schema_version                       | tokens
-------------+-------------+--------------------------------------+------+-----------------+-------------+--------------------------------------+-------------------------------------------
 10.8.25.113 |         DC1 | 0c11b261-7514-4c98-b94c-baa79c6c4faf | RAC2 |  1.2.0-SNAPSHOT | 10.8.25.113 | b438118e-baab-3212-a69d-6b091104523f |  {28356863910078205288614550619314017620}
 10.8.30.104 |         DC1 | ec7bf3be-41da-4cb5-8cec-63e5a8776bca | RAC1 |           1.2.0 |     0.0.0.0 | be91dd6b-8419-3f91-b5a0-86b071c1bfbe |                                      null
 10.8.25.134 |         DC1 | afdc7305-2a9e-471b-8a9c-5a27efd1423a | RAC2 |  1.2.0-SNAPSHOT | 10.8.25.134 | b438118e-baab-3212-a69d-6b091104523f | {113427455640312821154458202477256070480}
 10.8.30.103 |         DC1 | a6c4d3f6-dcbd-4801-aad3-ef0a26959e51 | RAC1 |  1.2.0-SNAPSHOT | 10.8.30.103 | b438118e-baab-3212-a69d-6b091104523f | {155962751505430129087380028406227096910}
 10.8.25.114 |         DC1 | a42f6766-279d-4d12-b549-d3d97c9ce650 | RAC2 |  1.2.0-SNAPSHOT | 10.8.25.114 | b438118e-baab-3212-a69d-6b091104523f |  {85070591730234615865843651857942052860}
 10.8.30.102 |         DC1 | 21b3af77-46e8-4814-8340-598617b57dad | RAC1 |  1.2.0-SNAPSHOT | 10.8.30.102 | b438118e-baab-3212-a69d-6b091104523f |  {99249023685273718510150927167599061670}
  10.8.30.14 |         DC1 | 24647d52-41eb-4df3-993e-51d4f841ca62 | RAC1 |  1.2.0-SNAPSHOT |  10.8.30.14 | b438118e-baab-3212-a69d-6b091104523f |  {70892159775195513221536376548285044050}
  10.8.30.15 |         DC1 | d84a5632-d6d5-4b06-8e1b-ae39ab185ca1 | RAC1 |           1.2.0 |     0.0.0.0 | 5cd8420d-ce3c-3625-8293-67558a24816b |                                      null
 10.8.25.132 |         DC1 | 71904dbb-9682-45a9-b9a9-c15f3e28c068 | RAC2 |  1.2.0-SNAPSHOT | 10.8.25.132 | b438118e-baab-3212-a69d-6b091104523f |  {14178431955039102644307275309657008816}
  10.8.30.16 |         DC1 | 73cf3d09-3c8a-41f0-a3fd-267e57216884 | RAC1 |  1.2.0-SNAPSHOT |  10.8.30.16 | b438118e-baab-3212-a69d-6b091104523f | {127605887595351923798765477786913079289}
 10.8.25.123 |         DC1 | 2d818a06-e62b-4027-9201-1801e885a446 | RAC2 |  1.2.0-SNAPSHOT | 10.8.25.123 | b438118e-baab-3212-a69d-6b091104523f | {141784319550391026443072753096570088100}
 10.8.25.133 |         DC1 | 940846e4-bb9c-4281-b583-7f76afa93589 | RAC2 |  1.2.0-SNAPSHOT | 10.8.25.133 | b438118e-baab-3212-a69d-6b091104523f |  {56713727820156410577229101238628035240}
  10.8.30.13 |         DC1 | b63fe173-5d13-4905-a59f-a78790f4f980 | RAC1 |  1.2.0-SNAPSHOT |  10.8.30.13 | b438118e-baab-3212-a69d-6b091104523f |                                       {6}

cqlsh:system> quit
root@scl-cas04:~# nodetool ring | grep 10.8.30.104
root@scl-cas04:~# nodetool ring | grep 10.8.30    
10.8.30.13      RAC1        Up     Normal  258.14 GB       8.33%               6                                           
10.8.30.14      RAC1        Up     Normal  542.67 GB       8.33%               70892159775195513221536376548285044050      
10.8.30.102     RAC1        Up     Normal  262.49 GB       8.33%               99249023685273718510150927167599061670      
10.8.30.16      RAC1        Up     Normal  288.43 GB       8.33%               127605887595351923798765477786913079289     
10.8.30.103     RAC1        Up     Normal  365.7 GB        8.33%               155962751505430129087380028406227096910     
root@scl-cas04:~# nodetool gossipinfo | grep 10.8.30
/10.8.30.14
  RPC_ADDRESS:10.8.30.14
/10.8.30.13
  RPC_ADDRESS:10.8.30.13
/10.8.30.102
  RPC_ADDRESS:10.8.30.102
/10.8.30.16
  RPC_ADDRESS:10.8.30.16
/10.8.30.103
  RPC_ADDRESS:10.8.30.103

more information as I debug but any thoughts about how this would be possible would be appreciated

Issue Links

• duplicates: CASSANDRA-5167 Node isn't removed from system.peers after 'nodetool removenode'