I0313 03:50:13.869735 136562 catalog_manager.cc:4076] T 7404240f458f462d92b6588d07583a52 P 7380d797d2ea49e88d71091802fb1c81 reported cstate change: term changed from 1 to 2, leader changed from 47af52df1adc47e1903eb097e9c88f2e (kudu-ts27) to 7380d797d2ea49e88d71091802fb1c81 (kudu-ts26). New cstate: current_term: 2 leader_uuid: "7380d797d2ea49e88d71091802fb1c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7380d797d2ea49e88d71091802fb1c81" member_type: VOTER last_known_addr { host: "kudu-ts26" port: 14100 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1952499f94a4e6087bee28466fcb09f" member_type: VOTER last_known_addr { host: "kudu-ts25" port: 14100 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47af52df1adc47e1903eb097e9c88f2e" member_type: VOTER last_known_addr { host: "kudu-ts27" port: 14100 } health_report { overall_health: UNKNOWN } } }
I0313 03:50:14.330202 136542 catalog_manager.cc:3828] Sending ChangeConfig:REMOVE_PEER on tablet 7404240f458f462d92b6588d07583a52 (attempt 1)
W0313 03:50:44.331295 136522 catalog_manager.cc:3216] TS 7380d797d2ea49e88d71091802fb1c81 (kudu-ts26:14100): ChangeConfig:REMOVE_PEER RPC failed for tablet 7404240f458f462d92b6588d07583a52: Timed out: ChangeConfig RPC to 10.75.5.28:14100 timed out after 30.000s (SENT)
I0313 03:50:44.331323 136522 catalog_manager.cc:3252] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 7404240f458f462d92b6588d07583a52 with cas_config_opid_index -1 with a delay of 56 ms (attempt = 1)
I0313 03:50:44.387584 136522 catalog_manager.cc:3828] Sending ChangeConfig:REMOVE_PEER on tablet 7404240f458f462d92b6588d07583a52 (attempt 2)
I0313 03:50:44.387991 136522 catalog_manager.cc:3637] ChangeConfig:REMOVE_PEER RPC for tablet 7404240f458f462d92b6588d07583a52 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER failed with leader 7380d797d2ea49e88d71091802fb1c81 (kudu-ts26:14100) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7380d797d2ea49e88d71091802fb1c81" member_type: VOTER last_known_addr { host: "kudu-ts26" port: 14100 } } peers { permanent_uuid: "d1952499f94a4e6087bee28466fcb09f" member_type: VOTER last_known_addr { host: "kudu-ts25" port: 14100 } } peers { permanent_uuid: "47af52df1adc47e1903eb097e9c88f2e" member_type: VOTER last_known_addr { host: "kudu-ts27" port: 14100 } }.
  Pending config: opid_index: 55447 OBSOLETE_local: false peers { permanent_uuid: "7380d797d2ea49e88d71091802fb1c81" member_type: VOTER last_known_addr { host: "kudu-ts26" port: 14100 } } peers { permanent_uuid: "47af52df1adc47e1903eb097e9c88f2e" member_type: VOTER last_known_addr { host: "kudu-ts27" port: 14100 } }
I0313 03:50:44.388002 136522 catalog_manager.cc:3252] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 7404240f458f462d92b6588d07583a52 with cas_config_opid_index -1 with a delay of 60 ms (attempt = 2)

I0313 10:16:18.844533 136648 catalog_manager.cc:4076] T 7404240f458f462d92b6588d07583a52 P 47af52df1adc47e1903eb097e9c88f2e reported cstate change: term changed from 2 to 3, leader changed from 7380d797d2ea49e88d71091802fb1c81 (kudu-ts26) to 47af52df1adc47e1903eb097e9c88f2e (kudu-ts27), now has a pending config: VOTER 7380d797d2ea49e88d71091802fb1c81 (kudu-ts26), VOTER d1952499f94a4e6087bee28466fcb09f (kudu-ts25), VOTER 47af52df1adc47e1903eb097e9c88f2e (kudu-ts27), NON_VOTER 08beca5ed4d04003b6979bf8bac378d2 (kudu-ts30). New cstate: current_term: 3 leader_uuid: "47af52df1adc47e1903eb097e9c88f2e" 
committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7380d797d2ea49e88d71091802fb1c81" member_type: VOTER last_known_addr { host: "kudu-ts26" port: 14100 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1952499f94a4e6087bee28466fcb09f" member_type: VOTER last_known_addr { host: "kudu-ts25" port: 14100 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47af52df1adc47e1903eb097e9c88f2e" member_type: VOTER last_known_addr { host: "kudu-ts27" port: 14100 } health_report { overall_health: HEALTHY } } } 
pending_config { opid_index: 54649 OBSOLETE_local: false peers { permanent_uuid: "7380d797d2ea49e88d71091802fb1c81" member_type: VOTER last_known_addr { host: "kudu-ts26" port: 14100 } attrs { replace: true } } peers { permanent_uuid: "d1952499f94a4e6087bee28466fcb09f" member_type: VOTER last_known_addr { host: "kudu-ts25" port: 14100 } } peers { permanent_uuid: "47af52df1adc47e1903eb097e9c88f2e" member_type: VOTER last_known_addr { host: "kudu-ts27" port: 14100 } } peers { permanent_uuid: "08beca5ed4d04003b6979bf8bac378d2" member_type: NON_VOTER last_known_addr { host: "kudu-ts30" port: 14100 } attrs { promote: true } } }


I0317 19:08:43.280254 136521 catalog_manager.cc:3828] Sending ChangeConfig:REMOVE_PEER on tablet 7404240f458f462d92b6588d07583a52 (attempt 6687)
I0317 19:08:43.280689 136521 catalog_manager.cc:3637] ChangeConfig:REMOVE_PEER RPC for tablet 7404240f458f462d92b6588d07583a52 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER failed with leader 47af52df1adc47e1903eb097e9c88f2e (kudu-ts27:14100) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7380d797d2ea49e88d71091802fb1c81" member_type: VOTER last_known_addr { host: "kudu-ts26" port: 14100 } } peers { permanent_uuid: "d1952499f94a4e6087bee28466fcb09f" member_type: VOTER last_known_addr { host: "kudu-ts25" port: 14100 } } peers { permanent_uuid: "47af52df1adc47e1903eb097e9c88f2e" member_type: VOTER last_known_addr { host: "kudu-ts27" port: 14100 } }.
Pending config: opid_index: 54649 OBSOLETE_local: false peers { permanent_uuid: "7380d797d2ea49e88d71091802fb1c81" member_type: VOTER last_known_addr { host: "kudu-ts26" port: 14100 } attrs { replace: true } } peers { permanent_uuid: "d1952499f94a4e6087bee28466fcb09f" member_type: VOTER last_known_addr { host: "kudu-ts25" port: 14100 } } peers { permanent_uuid: "47af52df1adc47e1903eb097e9c88f2e" member_type: VOTER last_known_addr { host: "kudu-ts27" port: 14100 } } peers { permanent_uuid: "08beca5ed4d04003b6979bf8bac378d2" member_type: NON_VOTER last_known_addr { host: "kudu-ts30" port: 14100 } attrs { promote: true } }
I0317 19:08:43.280700 136521 catalog_manager.cc:3252] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 7404240f458f462d92b6588d07583a52 with cas_config_opid_index -1 with a delay of 60045 ms (attempt = 6687)