[IMPALA-474] Statestore not properly detecting and evicting subscribers during failure leading to endless "Duplicate registration of subscriber: subscriber-id" messages

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.1
• Fix Version/s: Impala 1.1
• Assignee: Henry Robinson
• Reporter: Lenni Kuff
• Created: 20/Jul/13 14:52
• Updated: 16/Oct/13 16:07
• Resolved: 20/Jul/13 22:09

Description

This seems to happen when a subscriber takes a long time to processes/respond to a statestore heartbeat.

The expectation is that the subscriber would be detected as failed and removed as a registered subscriber by the statestore. The subscriber would then be able to re-register at some later point. Instead, it appears that the subscriber is unable to exit recovery mode and re-register because there is a duplicate subscriber with the same id.

statestore log:

I0720 07:49:27.306018  8037 state-store.cc:200] Registering: c2108.hal.cloudera.com:22000
W0720 07:49:27.306083  8037 state-store.cc:215] Duplicate registration of subscriber: c2108.hal.cloudera.com:22000, possible duplicate subscriber IDs or recovering subscriber
I0720 07:49:27.307083  8037 status.cc:44] Duplicate registration of subscriber: c2108.hal.cloudera.com:22000

impalad log:

I0720 07:40:39.108239 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:39.827592 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:40.609463 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:41.331050 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:42.112066 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:42.834488 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:43.613538 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:43.892231 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:40:43.894037 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:40:44.338031 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:45.116226 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:45.841644 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:46.617918 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:47.346148 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:48.120388 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:48.849720 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:48.925526 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:40:48.927409 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:40:49.622138 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:50.353080 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:51.124732 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:51.856823 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:52.625974 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:53.360152 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:53.967164 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:40:53.968950 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:40:54.128429 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:54.863665 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:55.630206 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:56.367118 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:57.132915 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:57.870497 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:58.634449 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:40:59.008869 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:40:59.010946 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:40:59.373926 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:00.136981 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:00.877496 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:01.638597 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:02.381037 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:03.141088 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:03.884661 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:04.050657 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:04.052602 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:04.642757 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:05.388032 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:06.156285 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:06.891603 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:07.657891 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:08.395002 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:09.100615 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:09.102507 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:09.160395 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:09.898450 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:10.661881 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:11.401840 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:12.164131 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:12.905462 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:13.665400 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:14.150589 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:14.152554 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:14.408906 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:15.167754 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:15.912247 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:16.669255 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:17.415946 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:18.171681 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:18.919355 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:19.200520 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:19.202378 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:19.673154 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:20.422845 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:21.175681 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:21.926602 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:22.677304 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:23.430042 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:24.179883 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:24.225621 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:24.227429 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:24.933720 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:25.681314 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:26.437294 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:27.184146 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:27.940613 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:28.685781 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:29.250594 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:29.270679 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:29.444113 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:30.188236 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:30.947736 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:31.689971 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:32.451208 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:33.192376 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:33.954684 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:34.275717 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:34.277498 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:34.694123 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:35.457928 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:36.196768 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:36.961437 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:37.698518 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:38.464917 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:39.201042 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:39.292350 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:39.294143 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:39.968437 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:40.702443 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:41.471871 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:42.204936 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:42.975334 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:43.706399 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:44.334002 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:44.335839 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:44.478919 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:45.208631 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:45.982328 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:46.710211 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:47.485772 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:48.212668 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:48.989140 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:49.359086 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:49.361450 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:49.714064 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:50.492615 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:51.217047 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:51.996052 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:52.718482 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:53.499519 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:54.220940 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:54.400701 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:54.402509 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:55.002266 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:55.724668 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:56.503984 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:57.228165 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:58.006587 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:58.731643 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:41:59.442508 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:41:59.444320 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:41:59.508056 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:00.235165 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:01.010912 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:01.738977 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:02.512183 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:03.242513 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:04.014757 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:04.484264 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:42:04.486007 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:42:04.746098 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:05.516217 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:06.249709 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:07.018688 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:07.753485 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:08.520226 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:09.257323 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:09.524435 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:42:09.526126 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:42:10.022763 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:10.760980 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:11.524358 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:12.264463 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:13.026993 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:13.767864 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:14.528444 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:14.559242 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:42:14.569003 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:42:15.271316 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:16.030956 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:16.775195 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:17.532444 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:18.278940 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:19.034843 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:19.569282 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:42:19.571109 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:42:19.782237 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:20.536229 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:21.285886 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:22.038895 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:22.789607 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:23.540736 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:24.293217 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:24.592574 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:42:24.594399 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:42:25.043479 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:25.796969 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:26.545081 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:27.300542 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:28.047665 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:28.804021 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:29.549072 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:29.617578 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:42:29.619318 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:42:30.307979 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:31.051813 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:31.811720 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:32.553163 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:33.315227 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:34.055994 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:34.659348 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:42:34.661175 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000
I0720 07:42:34.818675 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:35.557607 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:36.322190 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:37.060086 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:37.825713 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:38.561733 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:39.329226 32339 state-store-subscriber.cc:227] In recovery mode, ignoring update.
I0720 07:42:39.700930 32208 state-store-subscriber.cc:169] Trying to register...
W0720 07:42:39.702584 32208 state-store-subscriber.cc:181] Failed to re-register with state-store: Duplicate registration of subscriber: c2108.hal.cloudera.com:22000